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PREFACE 

This  document  was  prepared  by  Calspan  Corporation  under  Contract 
MDS903-76-C-0274.  The  contract  was  performed  for  the  Office  of  the  Assistant 
Secretary  of  Defense,  Office  of  the  Director,  Planning  and  Evaluation.  The 
project  was  monitored  by  Maj . George  Burkley. 

In  the  performance  of  this  contract,  Calspan  visited  and  inter- 
viewed personnel  of  many  government  agencies  and  private  companies.  The  co- 
operation of  these  organizations  and  the  individuals  interviewed  is  acknowledged. 

In  preparing  this  report,  Mr.  Wasserman  was  responsible  for  docu- 
menting the  literature  review.  Volume  II  of  this  report  is  based  on  the  infor- 
mation gathered  in  this  survey  of  existing  literature. 

Volume  I of  this  report  is  based  on  the  survey  interviews  per- 
formed primarily  by  Mr.  C.R.  Chalk. 

The  opinions  expressed  in  this  report  are  those  of  the  authors 
and  are  not  necessarily  shared  by  the  Department  of  Defense.  Publication  of 
this  report  to  document  the  study  and  to  promote  the  exchange  of  information 
and  ideas  in  no  way  implies  DOD  sanction  of  the  views  expressed  herein. 
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EXECUTIVE  SUWARY 

1.  This  is  a final  report  (for  the  period  15  April  1976  to  30 
September  1976)  on  the  Assessment  of  the  Role  of  Simulators  in  Military 
Tactical  Flight  Training,  based  on  a literature  survey  conducted  by  Calspan 
Corporation,  Buffalo,  New  York.  The  object  of  this  volume  of  the  report  is 
to  detail  the  data  obtained  relating  to  flight  simulation  (in  general)  with 
specific  attention  directed  towards  the  use  of  flight  simulation  devices  for 
tactical  flight  crew  training. 

2.  This  study  indicates  that  there  is  insufficient  objective 
data  available  with  which  to  perform  hard  trade  studies  to  analyze  various 
training  options,  or  to  demonstrate  the  cost-effectiveness  of  tactical  flight 
training  simulators.  The  cost-effectiveness  of  many  simulation  capabilities, 
e.g.  instructional  features,  is  not  well  documented.  Little  information  is 
available  on  the  impact  of  various  levels  of  "cue"  fidelity  or  "cue"  inter- 
action on  transfer-of-training.  The  existing  state-of-the-art  appears  to  be 
limited  in  its  capability  to  permit  substitution  of  simulator  hours  for  fly- 
ing hours,  except  primarily  for  procedural  (normal  or  emergency)  and  instru- 
ment flight  phases.  The  state-of-the-art  is  sufficient  to  provide  supple- 
mental training  in  the  tactical  mission,  provided  the  courses  are  tightly 
controlled. 

3.  Short-term  priorities  should  be  established  to  determine 
minimum  device  requirements  to  obtain  the  desired  training  objectives,  im- 
prove the  maintainability  and  reliability  of  existing  flight  simulators,  and 
evaluate  the  proper  training  and  utilization  of  instructors  to  provide  effi- 
cient and  effective  use  of  existing  facilities. 

4.  To  achieve  significant  reductions  in  the  flight  hours  re- 
quired for  military  tactical  mission  training,  continued  research  and  develop 
ment  is  required,  specifically  directed  at  visual  and  motion  system  require- 
ments and  mathematical  modeling  of  aircraft  (especially  at  flight  envelope 
extremes).  In  addition,  it  may  be  necessary  to  assess  the  application  of 
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parallel  digital  computer  concepts  and  modifications  to  existing  numerical 
solution  techniques  to  minimize  the  impact  of  digital  computer  limitations 
on  the  simulation  of  tactical  aircraft  and  representative  mission  environ- 
ments for  training. 

5.  Alternatives  to  the  use  of  simulation  to  conserve  limited 
fuel  resources  and  reduce  ordinance  costs  are  feasible  and  a detailed  assess- 
ment of  the  following  areas  is  recommended: 

e fuel  efficient  engine  development  for  aircraft 

e fuel  efficient  trajectories  using  optimal  control  theory 

e development  of  low  cost  airborne  instrumentation  system 
to  simulate  ordinance 

e development  of  low  cost  in-flight  simulation  capability 

e develop  more  effective  'wash-out'  screening  procedures 
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Section  I 

LITERATURE  SURVEY  INTRODUCTION 

The  objective  of  the  contract  performed  by  Calspan  was  to  examine  the 
application  of  specific  training  devices  (i.e.,  flight  simulators)  to  tactical 
flight  training  for  military  application.  This  application  assessment  would  be 
made  using  information  gathered  from  surveys  of  literature , industry  and  govern- 
ment agencies.  The  following  report  is  directed  at  the  literature  survey  phase 
of  this  study.  A survey  of  information  relating  to  flight  simulation  (in  general) 
was  performed  with  specific  attention  directed  towards  tactical  flight  crew  train- 
ing. It  should  be  said  at  the  on-set,  that  although  a significant  body  of  litera- 
ture relevant  to  flight  simulation  is  available,  the  quantity  of  information 
specifically  addressed  to  the  tactical  environment  is  relatively  sparse  (almost 
non-existent  for  operational  equipment) . Hopefully  this  is  not  a result  of  the 
technique  used  to  survey  the  existing  literature.  In  addition,  every  attempt  was 
made  to  be  objective  in  examining  the  literature,  however,  to  be  realistic  the 
subjective  biases  of  the  reviewers  tend  to  filter  through.  Thus,  a detailed  set 
of  references  specifically  used  for  this  study  will  be  listed.  The  reader  is 
welcome  to  examine  the  data  base  and  formulate  his  own  opinions. 

This  discussion  will  be  divided  into  the  following  specific  areas! 
First,  the  methodology  used  to  perform  the  survey  will  be  reviewed.  Data  ob- 
tained will  then  be  presented  as  applicable  to  general  areas  of  interest.  This 
will  be  followed  by  discussion  of  documents,  that  were  available  to  the  study, 
and  are  reasonably  specific  to  the  military  mission.  Finally,  the  section  will 
summarize  observations  made  from  the  literature  survey  concerning  the  present 
state-of-the-art,  short  term  vs.  long  term  alternatives,  technology  impact, 
and  review  various  alternative  solutions  to  achieve  specific  goals.  The  specific 
goals  are  extracted  from  Dr.  Allen's  article  appearing  in  Commanders  Digest 
(Reference  1).  However,  while  the  article  lists  *hese  goals  as  the  major 
objectives  for  developing  flight  simulators,  they  are  possibly  more  general, 
that  is,  they  are  the  broad  goals  of  a military  pilot  training  program.  These 
goals  are  listed  as  follows: 
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• reduce  costs  associated  with  training  and  operating 

• improve  and  maintain  the  skills  required  for  combat  readiness 
in  experienced  aircrew,  and 

• develop  better  pilots  and  aircrew  members. 

While  the  order  of  these  goals  presented  above  is  somewhat  inverted  from  the 
order  presented  in  the  referenced  paper,  it  should  be  noted  that  this  was  done 
intentionally.  It  appears  that  the  primary  driving  function  in  the  most  recent 
development  and  military  commitment  to  flight  simulators  is  basically  directed 
at  reducing  training  costs.  This  may  be  done  directly  by  substitution  of  "cost- 
effective"  simulator  training  hours  for  flight  time,  or  indirectly  by  supplemen- 
tary training  using  flight  simulators,  such  that  flight  training  hours  are  used 
more  effectively,  or  by  a combination  of  both  techniques.  However,  as  will  be 
shown  later  in  this  report,  the  requirements  imposed  on  the  state-of-the-art 
of  simulator  training  technology  will  be  influenced  by  the  choice  of  substitute 
or  supplement. 
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Section  II 

LITERATURE  SURVEY  METHODOLOGY 

Recently,  Calspan  Corporation,  under  Contract  to  the  B-l  Systems 
Project  Office,  performed  an  examination  of  the  Systems  Approach  to  Training 
for  the  B-l  aircraft.  Several  technical  reports  were  prepared  under  that  pro- 
gram among  them  was  the  Simulation  Technolog)'  Assessment  Report  (STAR) , Refer- 
ence 2.  This  document,  published  in  July  1975,  summarized  the  state-of-the-art 
in  both  engineering  and  behavioral  aspects  of  simulation  as  determined  for  that 
time  frame  and  to  some  degree,  as  applicable  to  developing  requirements  for  B-l 
training  device  systems.  This  publication  also  contains  an  extensive  bibliography 
of  documents  applicable  to  flight  simulation.  This  bibliography  has  served  as 
a " jumping-off"  point  for  conducting  the  literature  survey  required  by  this 
present  study  of  simulator  application  to  military  tactical  flight  training. 

The  time  frame  associated  with  the  present  contract  would  not  permit  an  extensive 
review  of  all  the  literature  available.  Thus  a technique  was  developed  in  an 
attempt  to  "optimize"  the  data  collection  process  within  the  time  and  bugetary 
constraints.  The  reports  used  for  the  B-l  SAT  Program  were  obtained  and  screened. 
The  screening  processes  consisted  of  the  following  steps: 

(1)  Report  was  classified  as  either  being  of  primary  or  secondary 
interest  to  the  study  (subjective  evaluation). 

(2)  Reports  were  screened  chronologically  and  reclassified  when 
more  pertinent  or  recent  data  were  obtained  . 

(3)  Reports  were  classified  as  to  subject  matter,  since  on 
certain  occasions  the  title  could  be  misleading  on  content. 

As  an  illustration  of  the  process,  it  became  obvious  during  the 
literature  survey  of  a chronological  progression  by  many  authors  describing 
various  concepts  or  devices.  Typically,  for  a device,  the  initial  report  or 
conference  paper  would  discuss  the  device  feasibility,  followed  by  a later  re- 
port describing  the  development  of  the  device  and  its  potential,  then  followed 
by  documentation  of  problems  in  the  integration  of  the  various  capabilities 
of  the  device,  and  finally  some  limited  information  on  the  utilization  of  the 
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device  for  training.  This  progression  also  exists  for  many  training  concepts. 
Thus,  any  assessment  would  therefore  tend  to  be  chronological.  It  was  therefore 
decided  to  attempt  to  concentrate  the  emphasis  of  detailed  examination  on  the 
past  six  years,  not  ignoring  the  previous  years,  but  primarily  relegating  that 
data  to  a background  role.  This  approach  was  deemed  desirable  also  because  of 
the  various  documents,  including  the  STAR  report,  which  have  reviewed  and  sum- 
marized much  of  the  past.  Since  the  STAR  document  was  based  on  literature  up 
through  and  including  1974,  it  was  necessary  to  extend  the  data  base  into  1976. 
This  was  accomplished  by  searching  the  primary  authors  and  sources  used  for  the 
STAR  document,  reviewing  recent  Calspan  acquisitions  lists,  cross-referencing 
literature  and  symposiums  reported  in  the  bibliography  of  recent  documents  and 
performing  computer  searches  using  Calspan  and  NTIS  facilities.  In  addition, 
recent  military/govemment  publications  (e.g.  Aerospace  Safety),  and  commercial 
journals  (e.g.  Aviation  Week)  were  surveyed  to  obtain  pertinent  literature.  By 
no  means  should  it  be  interpreted  that  an  exhaustive  literature  survey  was  con- 
ducted, since  this  would  have  been  outside  the  scope  (level-of-effort)  associated 
with  this  phase  of  the  contract.  No  doubt,  there  are  many  documents  presently 
available  that  were  not  obtained  for  this  survey,  or  which  could  not  be  used 
because  translation  of  foreign  language  reports  would  not  have  been  timely,  or 
that  were  in  the  publication  process  during  the  time  allocated  to  obtain  the  data 
base.  However,  it  is  the  opinion  of  the  author  that  the  data  base  collected 
from  the  literature  review  and  the  observations  made  from  this  information 
will  not  be  significantly  modified  by  a more  extensive  and  exhaustive  literature 
search.  Thus,  in  keeping  with  the  spirit  of  the  times,  the  literature  survey 
could  be  considered  to  be  "cost-effective." 
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FLIGHT  SIMULATION  FOR  THE  TRAINING  OF  MILITARY  TACTICAL  FLIGHT  CREWS 


Introduction 

This  section  is  based  on  the  data  obtained  from  the  literature  survey. 
Thus  the  observations  can  only  be  based  upon  that  information  which  the  various 
disciplines  and  organizations  involved  in  flight  training  via  flight  simulators 
are  willing  to  publish  or  present  at  various  meetings.  Data  and  opinions  obtained 
during  the  phase  of  this  study  that  involved  discussions  with  simulator  equipment 
manufacturers,  government  simulator  procurement  and  research  organizations,  and 
using  organizations  (both  military  and  commercial)  will  not  be  presented  in  this 
discussion.  This  section  will  be  organized  by  subject  matter,  however,  it  is 
extremely  difficult  to  organize  a discussion  in  this  manner  due  to  the  significant 
interaction  of  the  various  areas  of  interest.  Therefore,  some  redundancy  may 
become  apparent  while  reading  the  subject  material  to  follow;  every  attempt  has 
been  made  to  minimize  this  duplication  except  where  necessary  to  reinforce  perti- 
nent points  in  the  discussion.  The  intention  of  the  following  material  is  to 
present  observations  based  on  the  literature  survey  on  the  following  issues: 

• Simulator  Philosophy 

• State-of-the-Art 

• Training  Transfer 

• "Objective"  Performance  Measurement 

• Simulator  Acceptance  Testing 

• Cues  and  Fidelity 

• SAT/ISD 

• Instructor  Workload 

• Requirements 

• "Cost-Effectiveness" 

• Documented  Pilot  Opinions  on  Simulation 

(both  military  and  commerical) 

• Alternatives  and/or  Improvements  Affected  by  Technology 

• Attitudes,  Reliability  and  Maintenance 
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In  summary,  it  can  be  stated  unequivocally  that  there  are  insufficient 
data  available  with  which  to  perform  hard  trade  studies  to  evaluate  options. 

This  has  been  the  case  in  the  past,  and  to  some  degree  what  has  past  may  very  well 
be  prologue.  Several  significant  achievements  have  been  made,  or  are  in  the 
process  based  on  the  application  of  current  technology  to  simulator  equipment. 

As  an  example,  Computer  Generated  Imagery  has  made  significant  advances  in  the 
past  few  years,  however  it  is  not  quite  capable  yet  of  replacing  images  generated 
via  model  boards  (nor  is  there  any  reason  to  believe  that  such  a replacement  is 
either  necessary,  desirable,  or  the  most  cost-effective  solution).  However,  the 
questions  being  asked  about  visual  simulation  today,  are  still  essentially  the 
same  questions  as  were  asked  over  the  past  many  years  ( e.g.  how  much  field- 
of-view,  resolution  and  detail  is  required?) . Thus  to  some  degree  the  equipment 
used  for  simulation  has  become  more  complex,  sophisticated  and  hopefully  more 
reliable  ar.d  flexible,  however,  in  almost  every  case  the  fundamental  questions 
relating  the  application  of  simulation  to  flight  training  (especially  for  the 
specific  application  pertinent  to  this  study)  remain  relatively  unanswered. 
Simulation,  to  some  degree  like  magic,  is  an  attempt  to  create  illusion,  and  is 
still  fundamentally  an  art,  rather  than  a science.  Perhaps,  this  point  can  be 
illustrated  from  a paper  by  Johnson  and  Roscoe  (Reference  3),  entitled  "What 
Moves,  the  Airplane  or  the  World?"  This  article  published  in  1972  reviewed 
literature  written  for  the  past  36  years  pertaining  to  motion-relationship 
variables,  display  presentation,  and  the  effects  of  these  interactions  on  pilot 
performance.  Their  finding  was  significant  - "Experimental  findings  reported  range 
from  suspect  to  inconclusive.  The  best  experiments  from  the  standpoint  of 
scientific  rigor  have  been  conducted  in  fixed-base  simulators,  the  applicability 
of  which  is  suspect  in  questions  involving  physical  motion.  There  has  been  no 
conclusive  flight  experiment  dealing  directly  with  the  basic  issues  discussed." 

They  then  go  on  to  state,  " The  Questions  Must  be  Answered  in  Flight."  It  is 
our  impression  that  the  data  about  to  be  offered  in  this  section  will  lead  the 
reader  to  a similar  conclusion  concerning  the  application  of  flight  simulators 
to  military  flight  training  for  tactical  missions,  the  data  is  inconclusive  and 
answers  must  be  applicable  to  the  environment  in  which  the  threat  must  be 
treated,  the  pilot  must  be  capable  of  performing  his  function  in  the  aircraft 


in  the  air  in  a hostile  situation.  As  sinr.-ily  stated  by  Captain  W.A.  Woodin, 

CAA  and  J.D.  Cowell,  BOAC  in  Reference  4,  the  simulator  can  never  be  an  air- 
craft. An  aircraft  is  always  an  aircraft,  while  a simulator  attempts  to  behave 
like  an  aircraft.  There  are  both  advantages  and  disadvantages  to  this  unique 
situation.  The  remainder  of  this  discussion  will  attempt  to  summarize  the 
relative  advantages,  and  disadvantages  applicable  to  the  military  mission.  In 
addition,  it  will  be  hopefully  demonstrated  that  the  question  of  substitute  vs. 
supplement  is  quite  real.  Based  on  the  existing  information,  and  present  state- 
of-the-art,  supplementary  training  has  been  demonstrated  with  a certain  measure 
of  success  (TAC  ACES),  however  there  was  no  data  available  during  this  review  that 
could  justify  any  objective  decision  to  substitute  simulator  hours  for  flight 
hours  for  the  tactical  mission  especially  on  a routinely  operational  basis. 

Simulation  Philosophy 

As  a part  of  the  literature  survey,  it  was  decided  to  attempt  to 
examine  the  reasons  presented  in  the  literature  for  the  relatively  large 
military  and  commercial  commitment  to  simulator  utilization  for  flight  training. 

However,  it  may  be  important  to  first  attempt  to  list  those  attributes  of  a 

training  situation  which  allow  a learning  procedure  to  occur.  According  to  Drs. 
Huddleston  and  Rolfe  (Reference  5),  in  order  to  learn, the  following  ingredients 
are  required  (although  the  relative  importance  of  these  features  are  not 
discussed) : 

• The  individual  being  taught  must  possess  the  aptitude 
and  ability  to  learn. 

• The  individual  must  be  motivated  to  learn. 

• An  environment  where  practice  of  what  is  to  be  learned  must 

exist,  in  which  the  trainee  can  receive  feedback  which  identifies 
the  correctness  of  his  response. 

• A situation  where  his  performance  can  be  evaluated,  in  order 
to  demonstrate  that  learning  has  occurred. 


To  these  above  mentioned  items,  it  is  obvious,  that  the  attitude  of 
the  instructor  towards  the  features  and  conditions  of  the  learning  environment 
will  also  impact  on  the  process.  The  above  reference  further  goes  on  to  state 
that  while  the  flight  environment  can  provide  the  features  required  for  learning, 
it  also  introduces  other  features  which  are  restrictive  or  potentially 
undesirable.  These  additional  characteristics  are  classified  by  the  author  as: 


• Impossibility  (e.g.  gaming  area,  weather,  etc.) 

• Expense  (e.g.  0 § M costs,  weapons  cost) 

• Danger  (e.g.  training  accidents,  etc.) 


These  features  of  the  aircraft  then,  according  to  the  authors  help 
establish  a good  case  for  simulation.  Dr.  Allen  (Reference  1),  although  he 
lists  three  major  objectives  for  simulation  (develop  better  pilots,  maintenance 
and  improvement  of  combat  ready  skills,  cost  reduction)  maintains  that  the  pri- 
mary interest  of  the  DOD  in  modern  simulation  (if  properly  used)  is  the  develop- 
ment of  the  better  end  product,  rather  than  cost.  However,  at  the  same  time, 
he  also  states  (essentially  as  a blessing  in  disguise)  that  the  fuel  shortage 
of  1973  resulted  in  a sudden  increase  in  simulator  usage.  The  article  also 
lists  the  factors  which  have  influenced  the  emphasis  on  simulation  by  DOD  to 
replace  (substitution)  flying  time: 

• Development  of  new  aircrews  with  military  skills. 

• Maintain  or  enhance  those  skills  (military)  more  effectively 
in  experienced  aircrews. 

• Demonstration  of  the  "ideal"  maneuvers  and  procedures 

• Repeated  practice  in  short  time  with  instructional  features 
(e.g.  instant  replay). 

• Performance  evaluation  based  on  measurements  and  analysis 
by  simulator  computer  system. 

• Elimination  of  risk  in  practicing  edge  of  the  envelope  maneu- 
vers, or  in  practicing  tactical  maneuvers  in  a simulated  hostile 
environment. 

• Direct  savings  in  fuel,  and  other  operating  costs. 
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• Indirect  savings  including  wear  and  tear,  equipment  losses, 

personnel  losses,  reduction  in  bases  and  personnel  etc. 
required  for  training,  and  increase  effective  use  of  runways 
and  airspace. 

At  the  AIAA  11th  Annual  Meeting  in  Washington,  D.C.,  a paper  was 
presented  on  the  cost  effectiveness  of  simulation.  The  authors  of  this  paper 
state  " — economy  is  the  reason  for  simulation  in  the  first  place  — ."  For 
the  military  application  they  state  the  following  are  the  major  causes  for 
extensive  military  simulation  training  programs  (Reference  6) : 

• Economic  considerations 

• Safety  of  flight  training 

• Improved  flight  training 

With  the  last  two  factors  being  of  special  importance  for  emergency 
and  abnormal  situations. 

A very  direct  statement  concerning  the  subject  of  simulation,  was 
presented  by  J.  Nicholl  (Reference  7).  Although  he  lists  the  various  well 
known  reasons  for  simulation  (e.g.  training  improvements  arising  from  simulator 
facilities  not  available  in  the  aircraft  such  as  freeze)  he  goes  on  to  state 
"If  simulators  did  not  save  the  airlines  money  they  would  not  exist,  and  however 
valid  the  other  reasons  are,  we  would  have  no  difficulty  in  convincing  ourselves 
that  training  in  the  air  was  better  in  every  respect!"  Although  this  statement 
was  directed  at  the  airlines,  it  is  probably  much  more  general  in  scope. 

In  the  October  1974  issue  of  Aerospace  Safety,  Capt.  Jerome  T.  Bellen 
(Reference  8)  states  that  the  economic  reasons  for  simulation  would  be  sufficient 
justification,  and  he  proceeds  to  the  trimmings  such  as  safety.  In  addition,  he 
attributes  the  characteristic  of  "ultra  realism"  to  simulators.  While  it  is 
necessary  to  use  the  assumption  of  realism  to  some  degree  to  justify  the  desired 
impact  of  simulation  on  flight  training,  ultra  realism  may  be  a slight  (and  at 
this  point  in  the  state-of-the-art)  non-objective  exaggeration. 
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Major  General  Oliver  W.  Lewis,  USAF,  article  in  the  Air  University 
Review  (Reference  9)  essentially  tells  us  that  the  continuing  squeeze  of 
defense  dollars,  permits  only  one  option  - efficiency.  Training  stands  out  as 
an  area  because  of  the  high  individual  training  costs,  and  it  is  necessary  to 
make  flight  training  more  efficient.  The  article  then  continues  into  simulation 
as  the  new  approach  to  training  in  the  Air  Force.  To  quote  from  this  article, 
"Simulation,  properly  used,  stands  out  as  our  best  bet  for  simplifying  the 
learning  tasks  of  the  pilot  trainee  and  improving  the  quality  without  increasing 
the  cost  of  training." 

In  a recent  speech  to  the  AIAA,  Brigadier  General  Benton  K.  Partin 
USAF  again  emphasized  that  among  the  significant  reasons  for  the  increased 
interest  in  the  use  of  simulation  for  military  aircrew  training  is  lower  cost 
(both  from  fuel  savings  and  from  aircrew  training  costs) . However,  he  also 
recognizes  several  of  the  shortcomings  of  the  state-of-the-art,  essentially  the 
determination  of  training  requirements  and  what  is  the  effectiveness  of  the 
various  devices  for  training  various  tasks. 

Colonel  H.A.  Goodall  in  his  article  in  the  Air  University  Review 
"Simulation:  A Threat  to  Tactical  Air  Power,"  appears  to  be  a voice  for  the 
opposition.  He  discusses  the  recent  impression  that  because  some  training  was 
done  with  simulators  in  the  past,  that  now  many  feel  that  "cheaper  simulator 
hours  could  be  traded  on  a one-for-one  basis  with  more  expensive  actual  flying 
hours."  As  will  be  presented  later,  there  is  nc>  experimental  basis  for  such  a 
judgment,  especially  directed  at  complex  tactical  tasks.  The  appropriate 
transfer  experiments  have  not  yet  been  performed.  It  is  Colonel  Goodall's 
opinion  that  "the  savings  expected  from  simulation  in  tactical  mission  units 
portend  a potentially  serious  deletion  of  air  power."  He  then  goes  on  to 
state  "Simulation  should  be  viewed  as  supplementary  training  aimed  at  main- 
taining aircrew  skills  ."  Thus  simulation  "should  be  considered  addi- 

tive to  enhance  skills,  not  a trade-off."  This  appears  to  be  in  accordance 
with  the  recommendation  of  a Flight  Simulator  Symposium  held  in  January  1954 
and  referred  to  by  Dr.  Flexman,  et.  al.  in  Reference  10  "flight  simulation 
flying  time  is  not  a substitute  for,  but  rather  a complement  of,  airplane 
flying  hours." 


Essentially  then  the  die  has  been  cast.  The  primary  reason  for  the 
increased  emphasis  on  simulation  is  cost,  saving  limited  defense  dollars.  It 
is  indicated  by  the  proponents  of  the  use  of  simulators  that  not  only  will  they 
save  monies  but  also  produce  a better  pilot.  For  the  tactical  combat  mission, 
this  point  of  view  is  at  best  an  opinion,  unsubstantiated  by  experimental 
evidence.  It  then  appears  that  there  are  essentially  two  schools  of  thought 
on  the  use  of  simulation  that  are  emerging  (that  can  have  significantly  different 
impact  on  development  costs) : 

• Substitute  Training 

• Supplemental  Training 

If  the  primary  motivation  is  cost  reduction,  then  one  should  recognize 
that  alternatives  may  exist.  The  case  for  substitution  sometimes  refer  to  Apollo, 
however,  for  that  situation,  there  was  no  alternative  so  the  choice  of  simulation 
for  training  was  not  only  logical,  but  essentially  the  only  choice  possible.  This 
is  not  necessarily  true  for  military  training  if  the  primary  motivation  is  cost 
reduction.  The  fact  that  ordinance  use  can  be  simulated  does  not  necessarily 
restrict  this  capability  only  to  a ground-based  simulator.  Relative  to  the  use 
of  the  aircraft  for  training,  the  development  of  more  cost  effective  in-flight 
trainers  is  equally  possible,  especially  since  this  would  tend  to  negate  some  of 
the  monies  required  for  motion  and  visual  research,  the  application  of  modern 
control  theory  to  "optimize"  fuel  consumption  not  only  for  training  but  also  for 
routine  operational  flights  present  attractive  alternatives.  In  addition, 
training  economies  could  be  accomplished  by  utilization  of  a more  effective 
process  to  determine  pilot  candidates.  According  to  Dr.  A.  Diehl,  U.S.  Navy 
TAEG,  in  Reference  11,  "Approximately  25%  of  all  entering  candidates  fail  to 
graduate  as  Naval  Aviators."  He  suggests  that  the  inefficiency  in  screening 
candidates  for  military  undergraduate  aviation  training  could  be  improved 
using  a synthetic  selection  concept  which  requires  a relatively  inexpensive, 
unsophisticated  simulator.  A "wash-out"  minimization  program  could  possibly 
be  coupled  with  other  alternatives  reducing  training  costs  by  25%  without  the 
use  of  sophisticated  complex  simulators  and  their  required  support  facilities. 

In  addition,  the  supplementing  concept  could  be  used  with  relatively  less 
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complex  training  devices,  to  maximize  the  effectiveness  of  flight  training  hours. 
These  various  alternatives  for  cost  reduction  in  military  training  will  be  further 
discussed  in  other  sections  of  this  report. 

State  of  the  Art 

The  intent  of  the  following  discussion  is  to  review  the  state-of-the- 
art  in  training  simulation.  In  order  to  place  this  assessment  in  a proper  pro- 
spective, it  is  of  interest  to  review  those  questions  most  asked  as  to  what 
improvements  are  required,  or  in  what  areas  further  research  and/or  development 
are  deemed  necessary.  It  is  not  our  intent  in  this  discussion  to  summarize  the 
general  characteristics  of  simulation  devices,  this  is  most  satisfactorily  done 
in  the  STAR  report.  Reference  2.  In  addition,  the  STAR  report  includes  a review 
of  the  behavioral  aspects  of  simulation,  so  it  is  not  considered  necessary  to  re- 
peat this  in  depth.  Instead,  we  will  proceed  initially  with  a rather  historical 
approach.  This  will  be  accomplished  essentially  by  reviewing  the  recommendations 
that  have  been  presented  at  various  conferences  and/or  meetings  concerning 
simulation  in  general  and  simulation  for  training  aircrews  in  particular. 

In  August  1963,  at  Columbus,  Ohio,  the  AIAA  held  a meeting  on  simula- 
tion for  aerospace  flight.  The  following  observations  are  made  from  the  various 
papers  presented  at  that  meeting  concerning  what  were  considered  to  be  either 
problem  areas  or  significant  deficiencies.  The  approach  being  that  if  we  know 
this  information,  then  we  obtain  an  indirect  assessment  of  the  existing  state- 
of-the-art.  Rather  than  review  each  paper  in  detail,  the  following  presents  a 
summary  of  the  questions  raised  in  the  various  papers  and  studies  reported  at 
the  conference  (Reference  12) : 

• Visual 

• lack  of  resolution  inherent  in  T.V.  projection 

• lack  of  peripheral  visual  cues 

• perceptual  fidelity  priority  over  engineering  fidelity 

• pertinent  stimuli  in  appropriate  sequence 
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• requirements  for  design: 

• field-of-view 

• necessary  detail,  acquity 

• color 

• light  level  of  picture 

• mission  specific  details 

• time  lag  between  pilots  control  action,  visual  and 
motion  must  be  less  than  50-60  milliseconds. 

• Motion/Visual  Interaction 

• need  quantitative  data  on  contribution  of  motion  to 
improving  illusion  of  visual  display 

• need  data  on  cue  substitution 

• Motion 

• need  studies  to  define  influence  of  cockpit  motion  over  a 
wide  range  of  flight  tasks 
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• Computational  Requirements 

• digital  effects  on  numerical  stability 


• Model  Requirements 

• aircraft  representation,  minimize 

• Impact  of  Facility  on  Crew  Coordination  Training 

• improvements  in  performance  using  specialized  simulator 
training  only  significant  for  those  tasks  excluded  in 
actual  flight  practice,  not  learned  in  a classroom 
setting,  or  using  conventional  simulator  training 

• Fidelity  Determination 

• possible  use  of  "subjective"  pilot  rating/sensitivity  data 
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Essentially  the  principal  recommendations  resulting  from  this  con- 
ference appear  to  be  the  need  for  extensive  research  on  both  visual  display 
systems  to  provide  definitive  design  parameters  and  to  evaluate  motion  required 
or  cues  required  pertinent  to  the  selected  task.  Proceeding  now  to  a Symposium 
conducted  on  Flight  Training  Simulators  in  London  (Reference  13);  a review  of 
those  papers  indicate  the  following  simulation  deficiencies  or  problem  areas: 

• Substantial  improvements  required  in: 

• visual  attachments 

• motion  systems 

• reliability  and  maintainability  of  simulation  equipment 

• Visual 

• need  increased  field-of-view 

• need  improvement  in  resolution 

• need  improved  picture  quality 

• need  research  on  task  dependency  and  related  field-of-view 

• what  level  of  simulation  fidelity  is  required 

• what  cues  predominant  in  complex  task 

• need  to  establish  appropriate  requirements 

• need  research  on  how  visual  cues  interact/combine  with 
other  cues  to  form  the  perceptions  used  by  the  human 
operator  as  basis  for  decision 

• main  limitations  make  it  impossible  to  recreate  outside 
visual  scene 

• Motion 

• necessary  fidelity  to  meet  training  requirements  in 
different  flight  phases  is  a question 

• necessary  fidelity  to  obtain  aircrew  acceptance/satisfaction 

• further  advances  in  the  effectiveness  of  motion  systems 
requires  research  on  cue  interactions 

• need  accurate  motion  "threshold"  data 


14 


-i 


• need  reliable  and  non-spurious  motion  'cues' 

• what  number  and  which  degrees  of  freedom  are 
necessary  for  a particular  task 

Instructor 

• need  study  to  investigate  instructor  workload 

• qualifications  of  instructor  for  training  are  task  dependent 

• attitude  of  instructor  and  influence  on  success/failure 
of  training 

• proficiency  requirements  for  instructor 

• instructor  console  requirements/data  presentation 

• intelligent  use  of  device  and  device  features 

Performance 

• to  assess  must  first  be  able  to  define  task 

• inpact  on  various  types  of  information  on  pilot  performance 

• performance  may  not  be  appropriate,  perceptual  fidelity 
should  indicate  similar  responses  (control  strategy) 
with  flight 

• need  to  determine  just  what  a pilot  does  in  flight 

• need  study  to  replace  subjective  instructor  rating 
using  computed  performance  against  a standard 

Maintainability  and  Reliability 

• digital  computer  may  be  more  accurate  and  consistent 
but  not  necessarily  more  reliable  than  analog 

• high  serviceability  calls  for  inclusion  of  well-tried 
hardware  techniques;  R § D techniques  and  high  service- 
ability may  not  mix  under  operational  conditions 


• Motivational  Research 

• absence  of  fear  in  simulator  could  result  in  detraction 
from  training 

• what  does  "transfer"  really  mean 

• interaction  between  device,  student  and  instructor 

• Requirements 

• should  conduct  research  to  establish  minimum  consistent 
requirements  to  make  proper  compromises,  especially 
for  visual  systems 

• little  experimental  work  performed  to  date  to  determine 
the  level  of  simulation  necessary  for  adequate  transfei 

• motion  system  should  be  designed  to  provide  desired  infor- 
mation with  minimum  excursion 

• Cost-Effectiveness 

• increasing  simulation  system  complexity  to  attain 
increased  training  value  will  impact  cost-effectiveness. 
Assumptions  on  hours  and  operating  costs  very  significant 
on  any  trade  study 

Thus  between  1963  and  1970,  the  list  of  questions  grew  as  the  industry 
obtained  experience  with  the  complex  and  sophisticated  equipment.  The  next 
symposium  reviewed  was  the  AGARD  Conference  held  at  Ames  in  March  of  1970 
(Reference  14). 

Following  a panel  discussion, the  following  subjects  were  considered 
for  future  research  that  should  be  coordinated  by  AGARD: 

• Definition  of  existing  simulator  visual  and  motion 

limitations 

• Specification  on  simulator  capability  and  required  fidelity 

for  visual  and  motion  systems 


16 


' 


r? 


\ 


si 

i 

i 


• Research  in  Human  Perception 

• sensing  visual  and  motion  cues 

• relative  importance  of  cues 

• compensation  between  various  cues 

• modeling  of  physiological  system  and  pilot  behavior 

• Quantitative  methods  for  measurement  of  pilot  workload 

• Standardization  of  low  altitude  disturbance  model. 

Essentially  at  the  conclusion  of  the  meeting  at  Ames,  the  simulator 
people  gathered  felt  they  had  come  a long  way  since  the  early  "Link"  trainers 
however: 

(a)  considerable  work  remains  to  improve  visual  and  motion 
generation 

(b)  need  to  make  system  "cost-effective" 

(c)  Need  more  data  on  pilot  behavior,  since  he  is  the 
"inevitable"  judge  of  both  aircraft  and  simulator. 

Next,  let  us  list  the  conclusions/recommendations  from  a more  recent 
AGARD  Simulation  Conference  (Reference  15) . These  observations  are  listed 
below : 

• need  improved  math  models 

• need  better  prediction  of  aircraft  dynamic  derivatives, 
nonlinearities  and  unsteady  effects  at  high  angle-of-attack 

• need  to  improve/standardize  turbulence  model 

• need  to  improve  resolution  and  detail  of  computer  generated 
visual  displays 

• proper  generation  of  motion  cues  remains  a fundamental 
problem  due  to  limited  excursion  capability 

a need  more  information  about  human  vestibular  system  to 

improve  motion  drive  logic 

0 methods  used  to  measure  and  evaluate  pilot  workload/ 

performance  need  improvement 
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• most  elaborate  simulator  is  not  necessarily  'cost- 
effective'  - need  more  data  in  order  to  trade 

The  preceeding  historical  data  indicates  that  many  basic  questions 
related  to  the  state-of-the-art  of  the  application  of  simulators  remain 
unanswered.  There  are  strong  indications  of  technology  improvement,  however  it 
appears  for  example  that  one  is  still  concerned  with  required  resolution  and 
field  of  view  of  visual  systems.  Perhaps  the  most  significant  development  is 
that  in  1963  model  boards  generated  the  images  while  now  the  image  is  generated 
at  times  by  a computer.  That  the  current  state-of-the-art  is  somewhat  limited 
can  be  further  determined  from  the  list  of  problems  discussed  by  General  Partin 
in  his  address  to  the  AIAA  meeting  in  Dayton  recently.  The  specific  problem 
areas  mentioned  were  as  follows: 

• Task  Determination 

• Transfer  Ratio 

• Visual  Cues,  e.g.,  FOV,  Detail  and  Realism 

• Motion  Cues 

• Cue  Interaction 

• Simulation  of  Sensors 

• 'Objective'  Performance  Measurement 

• Aircraft  Data  on  Edge-of-the-Envelope 

• Cost  of  Motion,  CIG  and  Display  Systems 

• Simulator  Certification 

It  appears  then,  that  the  list  of  problems  with  simulation  remains  relatively 
extensive.  Finally  on  this  review  of  the  state-of-the-art,  there  is  a recent 
paper  co-authored  by  D.R.  Gum  and  R.R.  Swab  (Reference  16),  entitled 
"Technology  Challenges  in  Flight  Simulation."  This  paper  represents  possibly  the 
most  recent  and  general  assessment  of  current  simulation  technology  applicable  to 
military  aircrew  training.  Their  conclusion  is  that  "the  state-of-the-art  in 
many  areas  is  adequate  to  support  the  acquisition  of  effective  training  devices 
which  will  allow  considerable  aircraft  flight  training  time  to  be  saved." 

However,  an  examination  of  their  simulation  technology  assessment  table  indicates 
that  for  fighter  aircraft,  only  5 areas  out  of  20  were  considered  adequate,  while 
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4 of  20  were  considered  to  be  essentially  beyond  the  capability  of  present 
technology  even  for  initial  simulation  capability.  The  remaining  11  areas 
were  considered  to  be  feasible  but  required  either  performance  improvement  or 
cost  reduction.  Most  significantly  the  areas  considered  by  them,  to  be 
essentially  inadequate  at  present,  are  required  for  the  tactical  mission  (e.g. 
air-to-surface  and  low  level  navigation).  Thus  the  existing  state-of-the-art 
appears  at  first  glance  to  be  severely  restricted.  This  may  very  well  be  true, 
if  one  desires  to  use  the  simulator  as  a direct  substitute  for  flight  time  which 
appears  to  be  the  underlying  concept  used  by  Gum  and  Swab  in  their  paper. 

This  then  still  raises  the  question,  how  satisfactory  is  the  present  state-of- 
the-art,  not  to  substitute  for  flight  time,  but  to  supplement  flight  time? 

Data  to  be  presented  in  the  discussion  of  the  TAC  ACES  program  indicates  that  the 
latter  is  a lot  closer  to  short  term  realization  using  existing  hardware  capability 
then  the  former. 

Transfer  of  Training 

Inherent  in  the  justification  for  any  training  device  is  the  basic 
assumption  that  what  is  learned  in  the  simulator  can  be  used  in  the  aircraft, 
hopefully  in  a rather  positive  way.  In  order  to  quantize  such  a concept, 
especially  if  it  becomes  necessary  to  perform  cost-effectiveness  studies,  a 
set  of  calculable  parameters  have  been  historically  defined  and  used.  This 
set  of  quantities,  that  report  to  determine  the  effectiveness  of  the  device  are 
called  by  the  name  "Transfer  of  Training."  It  was  pointed  out  by  Hammerton 
(References  17  and  18),  that  basically  the  various  formulas  used  for  transfer 
can  be  classified  into  two  basic  groups: 

• 'measurement'  of  savings  in  training 

• 'measurement'  of  initial  performance  immediately  following 
the  transfer  to  the  real  world 

He  concludes  that  these  two  'measurements'  are  often  conflicting 
terms  of  device  evaluation,  and  which  measures  are  used  are  normally 
dependent  upon  the  need  of  the  user.  It  becomes  apparent,  that  for  specific 
tasks,  the  relative  importance  of  "first-shot"  transfer  may  be  very  important 
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to  the  use  of  simulators  for  tactical  training.  It  is  of  little  value  to  note 
as  Hanunerton  and  Tickner  do  in  Reference  19,  that  even  when  initial  transfer 
is  poor,  that  the  effect  may  be  very  transient  and  recovery  to  stabilized  per- 
formance proceed  at  a rapid  rate,  if  the  pilot  does  injury  to  himself  or  to  his 
vehicle  during  any  initial  performance  demonstration.  However,  again  in  this 
reference,  cost  considerations  appear.  The  authors  consider  that  the  practical 
significance  of  transfer  experiment  conclusions  are  clearly  dependent  upon  the 
impact  of  economic  considerations.  They  state  that  "good  initial  transfer  must 
be  weighed  against  the  increasing  cost  of  realistic  simulation."  In  Reference 
18,  it  can  be  observed  that  the  number  of  training  trials  in  the  simulator  are 
normally  neglected  in  the  determination  of  the  device  effectiveness.  This  point 
is  further  examined  by  S.N.  Roscoe  in  several  articles,  among  them  References 
20  and  21.  Although  these  references  indicate  that  extended  training  in  a 
simulator  can  produce  significantly  diminishing  returns,  the  true  value  of 
a simulation  device  must  include  such  additional  considerations  as  the  impact  of 
weather,  availability  of  aircraft,  etc.  That  could  make  it  advantageous  to  use 
a simulator  even  though  the  effectiveness  of  the  device  has  diminished  to  the 
point  that  it  is  no  longer  "cost-effective"  to  use  the  training  device.  This 
then  implies  that  when  the  choice  is  some  training  vs.  no  training,  the  choice 
is  for  some  training  although  that  specific  training  could  be  somewhat  ineffective. 

It  should  be  noted  that  inherent  in  a measure  of  training  transfer 
is  the  assumption  of  stabilized  performance.  Obviously,  to  measure  the  effective- 
ness of  a device,  one  must  be  able  to  define  a task,  and  a performance  index. 
However,  if  the  performance  index  is  ill-conditioned  then  the  true  value  of  the 
device  as  determined  from  'measurement'  of  training  transfer  is  somewhat  suspect. 
Unfortunately,  as  will  be  discussed  later,  there  appears  to  be  little  agreement 
on  the  "best"  way  to  combine  specific  activities  in  complex  situations  (i.e., 
aerial  combat)  to  yield  adequate  performance.  Reference  22.  Thus  when  a task 
becomes  complex  it  may  be  either  impossible  or  impractical  to  define  a meaning- 
ful measurement  of  performance.  Under  this  situation,  which  is  most  relevant 
to  tactical  flight  training  it  then  becomes  impossible  to  evaluate  the  device 
effectiveness  using  the  transfer-of-training  concept.  However,  in  Reference 
23,  the  authors  state  that  the  potential  for  the  modem  flight  simulator  to 
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produce  cost  savings  and  improve  the  quality  of  training  is  dependent  upon 
validated  transfer  of  training.  Thus  the  inability  to  describe  a performance 
reference  for  the  determination  of  device  effectiveness  (transfer  of  training) 
is  a significant  problem  area. 

Let  us  recall  that  the  purpose  of  the  "measured"  transfer  of  training 
falls  into  the  class  of  measures  that  can  be  used  to  determine  the  effectiveness 
of  a training  device  or  more  completely  a training  system.  An  obvious  goal  for 
a training  program  is  to  enable  the  student  upon  course  completion  to  be  able  to 
adequately  perform  his  required  tasks  in  an  operational  situation.  Using  the 
concept  of  the  Osgood  training  surface  (Reference  24),  then  the  training  device 
effectiveness  can  be  directly  related  to  the  level  of  response  similarity  to  a 
given  stimulus  both  in  the  aircraft  and  in  the  flight  simulator.  This  similarity, 
could  be  measured  relatively  to  a desired  performance  (providing  the  performance 
standard  is  definable)or  possibly  more  directly  by  the  similarity  of  control 
actions  between  the  simulator  and  the  aircraft. 

Before  examining  some  of  the  existing  literature  on  transfer  of 
training  results,  it  is  considered  necessary  to  raise  one  point  of  caution. 

It  has  been  suggested  in  several  references  that  terminal  performance  measures 
may  be  an  index  of  the  training  effectiveness  of  a device.  That  is,  for  example, 
probability  of  a hit  in  a weapons  delivery  task.  This  may  create  a pin-ball 
machine  effect  if  not  properly  used.  To  note  that  a students  time  on  target 
increases  with  training  in  the  simulator  (as  measured  in  the  simulator)  may  be 
nothing  more  than  a measure  of  the  pilots  adaptation  to  the  environment  of  a 
particular  training  device  and  should  not  be  considered  a measure  of  device 
effectiveness  unless  it  can  be  demonstrated  that  the  skills  learned  in  the 
simulator  can  be  readily  applied  to  the  actual  environment  without  extensive 
pilot  adaptation.  In  the  final  analysis,  the  pilot  must  perform  his  mission  in 
the  aircraft,  and  following  in  a logical  sequence  the  value  of  any  training 
concept/device/system  must  eventually  be  validated  in  the  operational  environ- 
ment, and  by  using  appropriate  tasks.  The  total  impact  of  the  training  ef- 
fectiveness of  a system  may  not  be  measurable  until  some  time  downstream.  This 
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requires  a post-performance  study  period.  An  illustration  of  this  situation, 
is  the  recent  newspaper  articles  that  are  indicating  that  the  "new  math"  has  become 
suspect.  What  was  reported  to  be  a significantly  improved  approach  to  the 
teaching  of  mathematics  may  have  created  the  situation  where  the  students  can 
easily  identify  whether  or  not  a number  may  be  greater  than,  or  less  than,  or 
related  by  set  theory  to  another  number,  however,  the  question  remains  for  cer- 
tain commercial  applications  "how  well  can  they  make  change  for  a dollar." 

This  post-performance  impact  of  simulated  training  was  also  reported  in  a Navy 
study  on  the  substitution  of  flight  simulator  time  for  aircraft  throughout 
instrument  training  in  the  Advanced  Jet  Syllabus  in  1972.  In  essence,  while  the 
performance  of  the  experimental  group  (simulator  trained)  and  the  control  group 
were  related  to  establish  transfer  ratio's  in  the  task,  subsequent  analysis  in 
more  advanced  stages  indicated  a statistically  significant  degradation  of  per- 
formance by  the  experimental  group.  This  then  implies  that  transfer  of  training 
is  highly  task  dependent,  strongly  device  dependent,  and  possibly  quite  fluid  or 
definitely  quite  iterative  and  may  only  be  truly  significant  at  the  time  of 
measurement.  If  we  quote  from  Reference  23,  the  authors  state  relative  to 
what  they  refer  to  as  Classical  Transfer  of  Training.  "The  classical  transfer 
approach  has  a place  in  the  initial  validation  that  particular  innovations  can 
be  effective,  but  it  probably  is  less  than  adequate  for  continued  assessment 
and  refinement  of  real-world  flying  training  programs.  It  is  too  cumbersome  to 
carry  out  and  often  the  results  are  too  specific  and  non-generalizable . A 
major  limitation  exists  in  the  difficulty  of  obtaining  sufficient  subjects  and 
maintaining  good  experimental  controls."  Thus  any  numerical  representation 
of  'measured'  transfer  should  be  qualified  and  caution  should  be  exercised  such 
that  generalizations  are  not  made  of  this  data  to  dissimilar  devices  or  training 
programs.  In  essence,  each  training  program  must  be  evaluated  on  its  own  merit. 

In  Reference  25,  S.  Mudd  indicates  that  there  are  several  limitations  that  can 
render  the  transfer-of-training  evaluation  concept  to  be  less  than  satisfactory: 

• Inapplicability  where  system  being  simulated  is  not 
yet  operational . 

• Inapplicability  where  simulated  system  is  so  complex  it 
may  not  be  feasible  to  use  an  "untrained"  control  group. 


• Approach  requires  each  new  device/concept  be  evaluated  for 
transfer  effects. 

• Technique  does  not  readily  provide  data  directly  for 
modifications  to  improve  the  performance  of  a given  device. 

In  addition  one  cannot  neglect  scheduling  problems,  and  motivational  problems 
in  attesting  to  obtain  transfer  data.  As  is  discussed  in  Reference  26,  the 
RAF  encountered  problems  in  attempting  to  perform  proper  streaming  trials.  In 
essence  the  experimental  crews  did  not  like  flying  additional  exercises  on  the 
simulators  while  their  counterparts  flew  aircraft.  It  is  further  interesting 
to  note,  that  this  cited  reference  is  describing  experiments  presently  being  con- 
ducted by  the  RAF  to  determine  the  effectiveness  of  training  devices  procured  six 
years  ago.  This  is  rather  typical  of  the  literature.  We  could  not  find  data  that 
could  be  used  to  properly  document  transfer  effects  for  tactical  military  train- 
ing. There  is  some  data  available  on  transfer  between  simple  simulators  and 
private  aircraft,  and  in  many  cases  for  procedual  and  instrument  flight  train- 
ing (e.g.  References  27  and  28  among  the  more  recent  data).  The  existing 
data  can  be  summarized  to  indicate  that  positive  transfer  can  be  obtained  from 
ground  based  flight  simulators  to  the  operational  environment  for  both 
relatively  skilled  or  naive  subjects  in  relatively  simple  flight  tasks  or  in 
training  for  procedual  tasks.  It  is  not  obvious  from  the  data  format  that 
stabilized  performance  was  obtained  prior  to  the  transfer.  The  data  does 
indicate  that  for  these  tasks,  and  using  relatively  simple  part-task  trainers, 
that  transfer  ratios  on  the  order  of  .250  to  .60  are  presently  within  the  state- 
of-the-art  of  simulator  training  systems.  Reference  29  indicates  that  a transfer 
ratio  of  .40  was  achieved  using  a relatively  simple  formation  flight  trainer. 

It  would  then  appear  that  a possibly  logical  and  potentially  cost- 
effective  approach  to  reducing  training  cost  while  improving  quality  would  be  to 
concentrate  on  improving  the  transfer  ratio  of  part-task  trainers.  One  could 
achieve  significant  savings  in  fuel  consumption  for  those  tasks  that  have 
already  been  demonstrated  as  highly  trainable  in  a relatively  simple  simulator 
rather  than  to  extrapolate  (with  significant  risk)  from  simple  tasks  to  highly 
complex  tactical  missions.  Obviously,  in  terms  of  judgment  as  well  as  analysis 


extrapolation  involves  more  risk  than  interpolation.  Further  thought  and 
research  is  required  to  establish  transfer-of-training  for  the  highly  skilled 
and  complex  tasks  associated  with  weapons  delivery  for  essentially  non-existent 
devices  for  routine  operational  usage.  If  it  is  deemed,  based  on  judgment, 
desirable  to  procure  advanced  (and  costly)  training  devices, then  the  building 
block  philosophy  used  by  the  Air  Force  Simulation  SPO  is  a logical  approach; 
however,  every  component  of  the  system  must  be  evaluated  for  its  impact  on  device 
effectiveness.  One  additional  cautionary  note  should  be  raised,  the  impact 
of  the  attitude  and  compentance  of  the  instructor  as  well  as  his  training  in 
the  effective  utilization  of  the  features  of  the  training  system  could  be  more 
significant  in  determination  of  training  effectiveness  than  many  of  the  physical 
characteristics  designed  into  the  system.  That  is,  as  has  been  stated  in  many 
references,  while  a good  instructor  can  compensate  for  a poorer  device,  a poor 
instructor  can  detract  significantly  from  a good  device  (e.g.  Reference  4).  As 
stated  in  Reference  30,  "Instructor  Pilot  quality  is  crucial  to  simulator 
training  effectiveness." 

Thus  perhaps  instead  of  using  transfer  experiments  to  determine  how 
much  simulator  flight  time  can  be  substituted  for  flight  time  or  just  how  com- 
plex does  a simulator  have  to  be  to  determine  the  relative  effectiveness  of 
training  with  simulators  , it  might  be  more  rewarding  to  examine  what  simulators 
have  already  accomplished  and  concentrate  on  improving  those  characteristics  to 
achieve  fuel  economy  in  training  and  reduce  flight  training  costs.  That  is, 
first  "optimize"  the  use  of  the  existing  state-of-the-art,  rather  than  to 
extrapolate  data  to  procure  devices  that  cannot  be  'objectively'  substantiated 
for  the  intended  mission. 

Training  Device  Concepts 

In  this  discussion  we  will  attempt  to  review  information  available 
on  several  concepts  associated  with  improved  "cost-effective"  utilization  of 
training  devices.  It  should  be  noted  that  again  the  literature  is  rather 
sparse  on  documentation  of  the  value  of  such  instructional  aids  as  problem 
'freeze',  adaptive  learning,  etc.  In  fact,  as  will  be  shown  in  this  section. 
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there  appears  to  be  more  warnings  on  the  use  of  special  instructional  features 
than  on  the  documented  validation  of  the  increase  in  training  effectiveness  for 
a device  equipped  with  these  special  features. 

The  following  list  presents  a majority  of  these  features: 

• problem  'freeze' 

• engagement  record/replay 

• reset 

• automated  flight  training  system 

• performance  monitoring 

e automatic  demonstration 

• hard  copy  printout 

• exercise  'speed  up' 

e automatic  malfunction  insertion 

• automated  opponent  ('iron  pilot') 

• instructor  station  layout 

For  this  discussion  it  would  seem  a good  idea  to  group  these  features  to  dis- 
tinguish between  those  devices  or  concepts  that  are  incorporated  to  save  time, 
and  those  devices  that  are  assumed  to  enhance  training  or  alternatively  reduce 
instructor  workload. 

Those  features  introduced  to  a flight  training  simulator  to  save 
time  are  essentially  'speed  up'  capability,  reset  capability  and  to  some  degree 
'problem  freeze'  although  the  latter  may  have  some  instructional  use.  Examina- 
tion of  the  literature  for  comments  concerning  these  features  reveal  that  the 
primary  remarks  addressed  to  this  capability  indicate  proceed  with  caution . 

That  is,  that  this  capability  is  considered  to  be  rarely  used  properly  and  when 
used  normally  destroys  the  illusion  that  the  trainee  is  in  an  aircraft  rather 
than  a simulator.  In  his  presentation  to  the  Royal  Aeronautical  Society, 

D.P.  Davis,  Chief  Test  Pilot  for  the  CAA  (Reference  31)  had  the  following 
comment : 
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• "Reset  is  useful  for  test  flying  simulators;  if 

its  use  for  line  pilots  can  be  avoided,  it  should  be." 

He  essentially  suggests  that  the  exercises  should  be  conducted  as  one  would  in  an 
aircraft.  With  respect  to  freeze,  he  considers  this  capability  as  "alarming" 

and  goes  on  to  suggest  "freeze  should  never  be  used  to  avoid  a crash " 

With  respect  to  exercise  'speed  up'  his  opinion  is  that  although  this  capability 
may  be  attractive  as  a time  saving  device,  this  feature  can  "clearly  destroy 
realism  and  should  never  be  used  with  regular  line  crews."  Although  his  com- 
ments are  directed  at  the  airlines,  they  are  essentially  more  general.  That 
is,  instructional  features  which  tend  to  destroy  or  significantly  modify  the 
realism  of  a training  situation  should  be  avoided  in  that  exercise.  This  does 
not  mean  that  such  features  may  not  be  of  value,  e.g.  in  setting  up  a problem  or 
initially  validating  a simulation  device.  The  opinion  expressed  is  concerned  with 
the  usefulness  of  these  capabilities  to  enhance  learning  by  destroying  reality. 
Similar  cautionary  advice  is  also  presented  by  Wooden  S Cowell  in  Reference  4. 

On  the  utilization  of  the  record/playback  capability  available  with 
modem  simulators,  the  views  expressed  in  the  literature  appear  to  be  much  more 
diverse.  For  example,  in  his  presentation  to  the  Second  Flight  Symposium  of  the 
Royal  Aeronautical  Society  in  May  1973  (Reference  32)  Captain  Thomas  of  UAL 
states  that  "We  have  found  that  most  of  the  instructors  do  not  use  the  record/ 
playback  capability  of  the  simulator;  rather  than  having  the  pilot  spend  time 
watching  an  approach,  they  much  prefer  to  have  the  pilot  fly  another  approach." 
Reference  33  indicates  that  the  instructor  pilots  used  in  the  reported  program 
carried  over  a flight  training  philosophy  to  the  simulator,  to  maximize  every 
minute  for  training.  However,  it  is  indicated  that  "replay  was  used  when 
sufficiently  important."  During  the  conducting  of  the  TAC  ACES  II  program 
(Reference  34)  it  was  determined  from  pilot  questionnaires  that  engagement 
replay  was  considered  as  the  most  valuable  instructional  feature,  however  the 
operation  of  this  system  is  complicated  and  could  not  be  consistently  used  in  a 
timely  fashion.  One  recommendation  of  the  TAC  ACES  II  program  was  simplification 
of  console  procedures  to  provide  rapid  and  accurate  record/playback  system 


operation.  Thus  it  appears  from  the  literature,  that  the  record/playback  feature 
can  be  a useful  instructional  aid  when: 

• it  does  not  interfere  with  the  training  exercise 

• it  is  simple  to  operate 

• it  can  be  used  in  a timely  fashion 

J.  H.  Goodier  (Reference  35),  states  that  the  record/playback  feature  "will 
enable  the  student  to  do  much  more  practice  on  his  own  and  will  relieve  the 
instructor  of  repetitive  work."  He  made  that  statement  in  1970,  and  to  date 
it  appears  that  this  goal  has  yet  to  be  achieved,  possibly  due  to  the  complexity 
of  the  record/playback  system  operation,  or  improper  training  of  the  instructor 
in  the  use  of  this  capability.  However,  we  should  not  overlook,  perhaps  from 
the  military  point-of-view  a highly  significant  aspect  of  the  record/playback 
capability,  that  is  for  debriefing  crews  after  a training  exercise.  As  discussed 
in  the  April  1976  issue  of  Aerospace  Safety  by  Captain  John  E.  Richardson  con- 
cerning the  use  of  ACMR  computer  capability  for  debriefing.  'The  display  gives 
the  student  total  recall  of  the  mission.  Each  maneuver  can  be  reviewed  and 
analyzed,  greatly  increasing  the  value  of  each  training  sortie."  Thus  in  review, 
there  does  appear  to  be  a consensus  that  the  record/playback  feature  can  enhance 
training.  Again,  it  must  be  stated,  that  there  has  been  no  experimental  evidence 
offered  comparing  the  training  effectiveness  of  a training  system  with  and  without 
the  utilization  of  this  capability.  In  addition,  through  the  use  of  pre-recorded 
tapes  it  is  possible  to  use  the  record/playback  capability  to  demonstrate  certain 
control  motion,  instrument  and  visual  (if  available)  relationships  that  exist 
when  the  exercise  if  flown  in  a "desirable"  manner,  possibly  obtained  by  the 
instructor  "flying"  the  training  device  and  recording  for  demonstration  purposes 
his  mission.  If  as  stated  by  Goodier  in  Reference  35,  that  there  is  an  opinion 
among  psychologists  "that  lessons  are  best  learned  when  there  is 
10%  explanation 
25%  demonstration 
65%  practice  " 

then  it  seems  reasonable  that  the  more  efficiently  the  explanation  and  demon- 
stration phase  can  be  accomplished,  the  more  effectively  the  device  can  be 
used  for  its  primary  teaching  role  - practice.  Again,  the  review  of  the  litera- 
ture has  primarily  indicated  the  feasibility  of  automated  demonstration  rather 
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than  the  effectiveness  or  efficiency  of  this  capability.  However  since  it  can 
be  obtained  almost  as  a "freebie"  when  a record/playback  capability  is  required, 
there  is  little  reason  to  attempt  to  determine  whether  or  not  a device  should 
have  this  feature  once  it  is  deemed  desirable  to  incorporate  the  more  extensive 
record/playback  feature.  The  question  that  still  remains  with  respect  to 
automated  demonstration,  is  really  when  and  how  to  use  it?  Additionally,  if  this 
capability  can  also  relieve  the  instructor  of  some  of  his  more  routine  workload, 
by  making  the  demonstration  phase  more  efficient,  this  in  itself  tends  to  indicate 
a potentially  desirable  feature. 


One  of  the  more  recent  developments  in  training  techniques  applied 
to  the  flight  simulator  is  the  apparent  emergence  of  automated  adaptive  flight 
training  systems.  Within  the  past  few  years,  assisted  by  the  application  of  the 
digital  computer  to  flight  training  systems,  automated  adaptive  training  has 
developed  from  a training  concept,  through  some  initial  feasibility  studies 
and  now  into  an  evaluation  stage  as  recently  discussed  at  the  NTEC  Conference, 
November  1975.  Prior  to  discussing  the  most  recent  results,  it  is  of  some 
value  to  review  the  concept,  therein  coming  to  a realization  of  the  requirements 
and  assumptions  involved  in  the  creation  of  an  automated  adaptive  teaching 
capability.  Fundamentally,  from  our  review  of  the  literature,  the  following 
definition  stated  by  Kelley  is  used  to  identify  Adaptive  Training  as  a concept 
within  which  either  "the  problem,  the  stimulus,  or  the  task  is  varied"  as  a 
function  of  performance.  Thus  as  indicated  in  Reference  36,  the  three  funda- 
mental elements  are: 

• task  variables  that  can  be  adjusted  to  change  the  level 
of  task  difficulty 

• a consistent  measurement  of  performance 

• a method  for  automatically  relating  task  difficulty  with 
performance,  such  that  the  level  of  difficulty  in  a task 
increases  as  skill  (measured  by  performance)  increases. 

That  reference  also  indicates  several  candidates  that  could  be  adjusted: 

• environmental  factor  (e.g.  turbulence) 

• operator  stress  (e.g.  modify  life  support  system) 


28 


• simulated  aircraft  variables  (e.g.  SCAS  gain) 

• feel  system  variables  (e.g.  backlash) 

• display  variation  (e.g.  predictor  display) 

• problem  generator  (e.g.  required  maneuvers) 

• task  loading  (e.g.  communications) 

While  the  author's  of  Reference  36,  suggest  the  above  list  for  its  "breadth 
of  choice"  it  is  obvious  that  many  of  these  potentially  adaptive  parameters  would 
not  necessarily  be  applicable  to  training  in  a specific  aircraft.  In  essence, 
the  primary  variables  used  in  feasibility  and  recent  evaluation  tests  normally 
can  be  classified  as  either  (a)  environmental  (wind  velocity  6 direction, 
turbulence),  or  (b)  stress  related  (i.e.,  emergency  conditions).  Critical  to 
the  use  of  an  adaptive  teaching  concept  is  the  ability  to  measure  performance, 
to  realistically  select  variables  to  modify  the  task,  and  to  insure  that  the 
task  difficulty  changes  introduced  do  indeed  enhance  the  learning  process. 

The  most  recent  applications  of  an  automated  adaptive  teaching  system  are  pre- 
sented in  References  37  and  38  respectively.  Essentially  what  has  been  demon- 
strated to  date,  is  that  potentially  the  adaptively  trained  student  is  equiva- 
lent to  the  conventionally  trained  student  for  the  automated  GCA  and  TACAN 
tasks.  However,  it  also  appears  that  there  is  still  much  to  learn  about  this 
concept  for  these  relatively  simple  tasks  for  which  reasonable  performance 
measures  are  possible.  The  extension  of  this  adaptive  concept  to  training  in 
the  tactical  environment  in  which  there  is  both  complex  skill  interaction  and  a 
diversity  of  opinion  on  what  truly  represents  adequate  and  measurable  performance 
(Reference  22)  is  presently  beyond  the  state-of-the-art.  Not  from  the  point 
of  hardware,  more  from  the  point  of  software  and  our  present  understanding  of 
skills  required  to  achieve  the  desired  performance  in  the  tactical  environment. 

It  is  this  very  inability  to  quantitize  the  "desired"  performance  on  a continuous 
basis  which  also  limits  the  application  of  transfer-of-training  analysis  to 
many  critical  military  tasks  to  evaluate  the  effectiveness  of  a training  device/ 
system.  Thus  if  adaptive  training  is  to  be  incorporated,  it  can  only  demonstrate 
its  "cost  effectiveness"  in  simple  tasks,  and  should  not  be  extrapolated  into 
more  complex  tactical  training  tasks,  since  a fundamental  requirement  (a  valid 
and  reliable  set  of  parameters  that  can  be  used  to  teach  to  desired  performance) 
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has  not  yet  been  determined  for  the  tactical  mission.  There  does,  however, 
exist  certain  terminal  or  scoreable  measures  (e.g.  time-on-target,  or  prob- 
ability of  hit)  of  performance,  the  question  appears  to  be  more  of  teaching  than 
just  by  sheer  repetition.  One  further  note  with  regards  to  automated  adaptive 
training,  it  should  not  be  used  in  any  way  that  would  tend  to  create  negative 
transfer  and  must  be  evaluated  for  each  intended  application.  At  present,  auto- 
mated teaching  has  been  used  in  the  GCA  tasks  which  has  also  incorporated  other 
features  into  the  training  system,  i.e.,  "voice"  system.  From  an  overall  point 
of  view,  since  the  services  are  still  presently  training  GCA  operators  it  may  be 
more  "cost-effective"  to  couple  this  training  together  than  to  provide  separate 
training  for  each  group.  Some  discussion  on  the  value  of  coupled  training  using 
simulation  is  discussed  by  Major  J.  Alwon,  RNLAF,  in  Reference  39. 

With  respect  to  hard  copy  printout,  this  feature  can  assist  in  both 
relieving  the  instructor  of  some  basic  bookkeeping  responsibilities  and  give 
a direct  indication  of  certain  parameters  to  the  pilot  trainee  to  examine  for 
progress.  However,  to  obtain  hard  copy  should  not  inconvenience  or  restrict 
tne  training  exercise,  as  indicated  in  Reference  33.  Fundamentally  to  serve 
as  an  aide  to  training,  it  is  the  timeliness  of  the  data  that  is  of  fundamental 
importance.  If  the  student  must  wait  several  days  for  off-line  processing  of 
dat^  its  value  as  feedback  is  essentially  minimized  if  not  eliminated,  however 
this  off-line  data  processing  could  still  satisfy  a requirement  to  "track"  the 
students  performance.  Direct  on-line  processing  will  have  both  hardware  and 
software  implications.  While  there  is  evidence  that  immediate  feedback  of  per- 
formance in  a task  can  improve  subsequent  performance  in  that  same  task 
(Reference  40),  there  is  no  data  to  indicate  that  the  student  reviewing  a 
hard  copy  is  any  better  than  the  insturctor  telling  the  student  for  example  what 
his  rate  of  descent  was  at  touchdown.  There  is  evidence  however  that  numerical 
information  on  terminal  performance  parameters  may  have  an  increased  impact  on 
subsequent  performance  than  the  use  of  general  descriptive  phases,  as  discussed 
by  D.B.  Armstrong  and  G.  Musker  in  their  paper  presented  at  the  Royal  Aero- 
nautical Synqposium  on  Flight  Training  Simulator's  (Reference  41). 
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While  there  is  little  doubt,  that  the  insertion  of  malfunctions  either 
manually  by  the  instructor  or  automatically  by  a predetermined  set  of  conditions 
should  be  part  of  a training  simulator  program  (especially  since  one  of  the 
arguments  in  favor  of  simulation  is  reduction  of  risk  in  teaching  emergency 
procedures,  etc.)  the  more  fundamental  issue  is  how  many  are  of  value  to  the 
desired  training.  As  discussed  in  Reference  32,  recent  training  devices  are 
required  to  store  more  malfunctions  than  are  required  to  complete  the  training 
syllabus.  This  then  indicates  an  inefficient  use  of  computer  storage  and  a 
potential  waste  of  programming  effort  with  of  course  the  additional  costs  and 
demonstration  requirements  for  acceptance.  It  then  seems  logical  that  by 
evaluating  the  desired  level  of  proficiency  and  with  some  knowledge  of  critical 
malfunctions  associated  with  certain  aircraft,  malfunction  insertion  can  be  made 
more  effective.  In  addition,  it  could  be  highly  undesirable  in  terms  of  crew 
motivation  to  use  unrealistic  malfunctions,  or  improper  sequencing  of  mal- 
functions . 


A recent  training  feature  discussed  in  the  literature  for  air-to- 
air  combat  training  is  the  use  of  either  a pre-programmed  target  using  a canned 
maneuvering  target  (e.g.  TAC  ACES  I)  or  programmable  interactive  threat  air- 
craft (Reference  42).  Another  technique,  which  is  discussed  in  Reference  33, 
used  by  Northrop  was  to  develop  the  capability  for  the  instructor  to  have  a 
stick  steering  mode  which  gave  him  limited  bogey  control.  During  the  TAC 
ACES  I program  it  was  determined  that  a "pre-recorded  maneuvering  target  feature 
is  of  limited  utility."  The  computerized  "hostile"  described  in  Reference  42 
and  the  techniques  presented  in  Reference  33  indicate  the  desirability  of  bogey 
control  during  a training  exercise.  There  is  no  documentation  in  the  literature 
surveyed  that  the  observed  increase  in  performance  in  the  air-to-air  combat 
role  using  a controllable  "bogey"  in  simulation  exercises  transfers  to  the 
operational  environment.  However  it  must  be  remembered  that  improper  training 
against  a potential  threat  is  highly  undesirable  and  possibly  disastrous 
(Operation  Rolling  Thunder),  as  discussed  in  International  Defense  Review, 
Reference  43. 
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With  respect  to  the  features  desired  or  required  for  the  Instructor/ 
Console  Operator  Station,  it  becomes  readily  apparent  that  this  feature,  as 
are  many  others  in  the  field  of  flight  simulation,  is  still  under  development. 

To  quote  from  Reference  33,  " The  instructor's  console  development  was  by  far 
the  most  technically  challenging  aspect  of  the  entire  simulation  program." 
Discussions  at  the  Flight  Training  Symposium,  held  by  the  Royal  Aeronautical 
Society,  October  1970  indicate  that  there  was  "no  knowledge  of  an  investigation 
into  the  workload  of  the  instructor."  This  point  is  somewhat  repeated  by 
Captain  Thomas  in  his  paper  presented  in  1973  (Reference  32).  "Much  research 
has  gone  into  the  design  of  automatic  malfunction  insertion,  automatic  feedback, 
record/playback,  demonstration  capabilities  and  other  devices  to  make  the 
instructor's  job  easier  and  more  efficient.  Yet  it  appears  that  very  little 
effort  has  gone  into  finding  out  exactly  what  the  instructor's  job  is  " "In 

an  effort  to  relieve  the  instructor  of  some  of  the  time  consuming  tasks , 

we  have  provided  him  with  considerable  trainer  capability  which  he  does  not 
use."  The  author  considers  the  application  of  the  behavioral  objective  concepts 
to  determine  what  the  instructors  job  is  of  equal  importance  to  the  application 
of  this  methodology  to  the  determination  of  aircrew  functions  required  to  per- 
form the  task.  Throughout  the  literature,  one  can  find  statements  which  relate  to 
the  importance  of  .he  instructor,  the  training  requirements  and  the  capabilities 
required  for  a good  instructor.  To  quote  Ralph  E.  Flexman,  " the  instructor 
has  been,  and  always  will  be,  the  most  important  variable  in  any  training 
situation."  The  impact  of  the  instructor  is  also  most  recently  discussed  in 
the  Summary  Report  of  the  Task  Force  on  Training  Technology,  Reference  44. 

However,  in  this  reference,  on  the  content  dealing  directly  with  flight  training 
there  continues  to  appear  the  message  that  the  development  of  innovative  concepts 
and  strategies  will  lead  to  improvement  of  the  cost-effectiveness  of  existing 
simulators  and  trainers.  This  may  well,  fall  somewhat  into  the  area  of  a 
desirable  goal  rather  than  a measurable  goal.  In  agreement  with  the  author 
of  that  section  (Dr.  Flexman)  the  literative  review  does  indicate  that  more 
energy  has  been  devoted  to  design  and  development  of  facilities,  than  on 
teaching  strategies,  obviously  since  there  seems  to  be  limited  data  at  best  on 
defining  the  job  of  the  instructor.  However,  to  some  degree,  it  is  necessary 
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to  have  devices  with  which  to  perform  the  required  experiments  in  a logical 
scientific  manner,  if  one  is  attempting  to  gather  data  on  the  "cost-effectiveness" 
of  an  innotative,  or  for  that  matter,  a more  conventional  teaching  strategy 
or  device  or  a training  system.  This  report  (References  44)  goes  on  to  state 
as  a general  observation  that  "There  are  no  cost-effectiveness  functions  that 
would  permit  the  comparison  of  the  current  with  alternative  training  philosophies, 

methods,  procedures,  equipments  and  goals The  implication  is  both  clear 

and  disturbing many  important  decisions  are  being  made  without  adequate, 

valid,  quantitative  basis  in  fact."  This  statement  again  indicates  that  it  is 
necessary  for  the  simulator  industry,  the  engineering  psychologists,  and  the 
military  user  to  sit  down  and  use  the  existing  facilities  to  do  their  home- 
work first  to  attempt  to  obtain  the  data  required  to  evaluate  military  flight 
training  systems. 

Thus,  based  on  the  literature  review  concerning  training  device 
instructional  features  and  application  of  these  features,  one  comes  away  with 
the  impression  that  there  presently  exists  no  quantitative  data  that  estab- 
lishes the  cost-effectiveness  of  these  various  specialized  instructional  fea- 
tures. There  is  some  opinion  that  such  capabilities  should  not  be  used  to 
disturb  the  "realism"  of  a training  exercise,  and  that  the  operation  of  such 
features  should  be  timely  and  uncomplicated.  The  record/playback  feature  if 
properly  used,  could  provide  one  of  the  desired  characteristics  of  a training 
system,  immediate  feedback  to  the  trainee.  At  the  October  1976  meeting  of  the 
A1AA  Working  Group  on  Training  Simulation,  B.  Cream  of  Human  Resources  Labora- 
tory, Wright-Patterson  Air  Force  Base  outlined  a recently  initiated  two-year 
program  to  systematically  evaluate  cost/training  effectiveness  of  simulators 
and  instructional  features.  The  successful  completion  of  this  program  would 
hopefully  answer  many  of  the  questions  raised  in  the  preceding  discussion. 
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Systems  Approach  to  Training  (SAT) 

Perhaps  one  of  the  more  innovative  strategies  recently  extended 
to  the  flight  training  task  has  been  the  attempt  to  apply  the  concepts  of  sys- 
tem theory.  This  approach  normally  referred  to  as  SAT  or  ISD  (instructional 
systems  development)  has  received  some  advanced  publicity  as  a technique  to 
"ensure  that  the  most  cost-effective  training  methods  and  equipment  will  be 
developed  and  processed."  However,  as  Reference  44  indicates,  "There  is  some 
question  regarding  the  degrees  to  which  it  is  being  correctly  implemented,  and 
there  is  little  or  no  evidence  of  its  actually  producing  cost-effective  re- 
sults." This  statement  is  reinforced  in  Reference  23  concerning  the  output 
of  ISD  teams  to  develop  new  flight  training  curricula  and  syllabi.  "In  some 
instances,  the  end-product  has  been  a less  than  optimal  use  of  the  technology 
(ISD);  several  of  these  early  programs  were  little  more  than  adaptations  of 
previous  programs  and  perpetuated  many  of  the  existing  inefficiencies."  This 
| reference  then  goes  on  to  state  what  are  limitations  of  the  approach: 

• Dependency  on  initial  analysis  such  that  inaccuracies  can 
be  locked  in  and  inappropriate  training  program  defined. 

• Analysis  must  be  performed  by  "trained"  experts  in  the 
methodology . 

It  appears  therefore  that  this  concept  may  have  numerous  difficulties  in  the 
application  stage,  especially  since  the  existing  data  base  on  task  features 
required  to  be  trained,  as  well  as  the  minimum  device  requirements  to  provide 

| . ‘ 

such  training  is  essentially  non-existent  for  many  tasks.  Thus  there  appears 
to  be  a strong  potential  for  GIGO  (Garbage  In,  Garbage  Out).  Again,  as  the 
authors  of  Reference  23  reiterate,  SAT  approaches  are  complex  and  abound  in 
* the  opportunity  for  error.  Obviously  a major  source  for  error  is  in  neglect- 

ing  a portion  of  the  task  to  be  trained.  The  Specific  Behavioral  Objectives 
(SBO)  concept,  within  the  systems  approach  essentially  assumes  that  the  task 

can  be  well  enough  defined  that  it  can  be  properly  sequenced  and  that  one  can 

i 

determine  "what  is  required  to  accomplish  the  task",  rather  than  "what  is 

| 
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nice  to  know."  Thus  as  discussed  by  Cream,  et  al  in  Reference  45,  the  behav- 
ioral data  acquisition  phase/process  "assumes  that  all  tasks  can  be  reduced 
into  more  detailed  subtasks  and  that  task  descriptions  can  be  in  the  form  of 
S/R  (Stimulus/Response)  descriptions".  The  application  of  this  assumption  to 
some  of  the  less  complex  tasks  (e.g.  procedural)  may  be  quite  valid,  however, 
the  application  is  not  quite  as  clear  for  some  highly  complex  duties."  Unfor- 
tunately, for  military  tactical  training,  the  tasks  can  require  complex  and 
highly  interactive  human  skills  and  it  may  be  extremely  difficult  to  develop 
an  acceptable  set  of  behavioral  objectives.  Reference  45  then  goes  on  to  dis- 
cuss a new  methodology  for  the  determination  of  specific  task  data  and  the  inter- 
action of  behavioral  data  in  the  design  of  training  devices.  This  new  method- 
ology is  essentially  based  on  a subtractive  concept.  That  is,  assume  that  one 
can  include  all  the  capability  in  the  trainer  that  is  required,  and  then  inter- 
act with  the  device  user  to  delete  unnecessary  capability.  This  concept  is 
somewhat  opposite  to  the  task  analysis  performed  by  the  engineering  psycholo- 
gist to  determine  specific  behavioral  training  objectives,  however,  in  essence, 
both  approaches  are  complementary.  Perhaps  the  strongest  issue  for  the  approach 
suggested  by  Reference  45  is  its  dependency  on  interaction  and  iteration 
with  the  user  for  the  justification  of  device  capability.  An  extensive  utili- 
zation of  SAT/ISD  methodology  for  specific  application  to  developing  a train- 
ing system  for  the  B-l  aircraft  was  recently  completed  by  Calspan  Corporation, 
Reference  2.  In  this  document,  the  authors  again  caution  that  SAT  is  a 


"decision  aid"  not  a panacea.  This  report  again  reiterates  a significant  prob- 
lem with  the  SAT  approach,  that  is,  "the  lack  of  quantitative  relationships 
among  the  factors  that  affect  the  system."  Thus,  it  appears  that  for  the  near 
term,  the  SAT  approach  for  many  tasks  associated  with  military  training  sys- 
tems will  do  nothing  more  than  quantitize  the  relative  merits  of  alternative 
training  concepts  based  upon  subjective  bias  of  the  group  applying  the  method- 
ology rather  than  using  objective  quantitative  data  which  is  required  to  pro- 
perly apply  the  systems  concept.  Thus  again,  it  appears,  from  examination  of 
the  various  recent  discussions  concerning  SAT,  that  to  minimize  the  potential 
risk  of  extrapolation  of  an  inadequate  or  improper  data  base,  the  methodology 
should  only  be  used  where  the  data  has  been  established  with  reasonable  cer- 
tainty and  reliability.  It  is  of  little  value  to  apply  SAT  concepts  if  the 


i 


i 


decisions  are  actually  being  based  on  judgement  anyhow,  to  establish  the  value 
of  alternative  training  concepts/systems . In  addition,  if  it  is  as  truly 
complex  to  apply  the  SAT  methodology  as  we  are  lead  to  believe  from  Reference 
23,  and  if  applied  with  inappropriate  initial  data  can  lead  to  specifying  the 
wrong  device,  training  program,  etc.,  then  one  must  consider  this  approach 
with  extreme  caution.  The  determination  of  appropriate  data  should  logically 
precede  the  application  of  ISD  to  the  procurement  of  new  training  systems. 

At  the  present,  if  we  must  rely  on  judgement,  then  it  is  essential  that  the 
ultimate  user  must  significantly  interact,  on  a continuous  basis,  with  the 
personnel  responsible  for  development  and  procurement  of  training  systems,  and 
vice  versa.  Perhaps  a reasonable  solution  is  to  follow  the  Navy  example  and 
establish  TAEG-type  units  as  recommended  by  Ralph  Flexman  in  Reference  44. 

This  could  then  provide  a framework  to  continually  analyze  and  evaluate  the 
application  of  training  devices  to  a specific  task  and  if  nothing  more  could 
sharpen  the  judgement  required  for  future  procurement  decisions. 

Maintenance  and  Improvement  of  Skills 

Earlier  in  this  report.  Dr.  John  L.  Allen's  article  (Reference  1) 
was  cited  which  indicated  that  one  of  the  major  objectives  in  the  pursuit  of 
flight  simulators  was  the  "maintenance  and  improvement  of  combat  readiness  in 
experienced  aircrews."  Thus,  it  was  decided  to  explore  the  recent  literature 
for  data  on  skill  retention,  upgrading  using  simulators,  and/or  "proficiency" 
training.  Again,  it  was  not  possible  to  exhaustively  search  the  literature, 
however,  several  documents  were  found  that  are  relevant  to  this  subject. 

Reference  46  reports  an  experiment  on  forgetting  instrument  skills. 
The  authors  indicate  at  the  time  the  study  was  conducted  (1971)  that  "there  is 
little  objective  data  on  the  rate  and  amount  of  forgetting  of  several  response 
classes  that  comprise  flying."  The  results  of  the  study  performed,  using 
subjects  "naive  to  flying"  were  the  following: 

• Discrete  procedural  responses  are  easier  to  forget  than 
flight  control  responses, 
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• Retention  loss  was  not  influenced  by  the  amount  of  initial 
training, 

• Relearning  procedures  is  dependent  upon  the  amount  of  ini- 
tial training. 

The  implication  of  this  study  is  that  for  operational  organizations  it  could 
be  effective  to  provide  relatively  uncomplex  training  devices  to  maintain 
procedural  response  skills. 

There  are  several  articles  also  written  which  indicate  that  a 
majority  of  users  feel  that  simulators  are  only  valid  for  emergency  and  pro- 
cedural training.  This  was  reported  as  recently  as  1973  by  a survey  con- 
ducted by  the  British  Airline  Pilots  Association  PEL/Training  Study  group. 

Data  on  the  Airline  Pilots  attitude  toward  simulation  in  this  country  was  not 
uncovered.  An  article  recently  did  appear  in  Professional  Pilot,  June  1976, 
by  John  Lowery  of  FSI . The  impact  of  this  article  is  that  simulators  can 
and  have  indicated  their  capability  to  increase  "proficiency  while  reducing 
training  costs."  The  author,  while  directing  his  attention  to  essentially 
crew  recurrency  training  suggests  that  "discussing  it  or  touching  the  appro- 
priate levers  is  only  slightly  better  than  not  practicing  at  all."  It  is  also 
interesting  to  note  in  the  June  30,  1976  issue  of  the  Convention  News,  in 
which  FSI  announces  their  new  digital/visual  Falcon  20  simulator,  the  new 
annual  fee  for  FSI  refresher  training  was  increased  by  33-1/3%. 

In  the  literature  examined,  little  data  is  available  on  the  use 
of  ground  simulators  for  the  improvement  of  combat-ready  skills.  The  only 
statements  uncovered  were  related  to  the  TAC  ACES  programs  (References  30 
and  34)  which  indicate  that: 

• Pilot  subjects  considered  programs  would  be  of  greater  bene- 
fit to  the  inexperienced  ACM  pilot  than  to  the  experienced 
pilot. 
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• Necessary  Air/Air  skills  "cannot  be  honed  in  the  simulator." 
Cue  Fidelity  Requirements 

It  is  not  the  intent  of  this  discussion  to  provide  a complete 
review  of  motion  cues,  visual  cues,  aural  cues,  etc.  and  their  interaction. 
There  are  many  reports,  articles,  etc.  available  which  address  themselves  to 
that  task.  Instead,  it  is  the  'r.tent  of  the  following  to  examine  what  is 
known  about  the  impact  of  these  cues  on  training,  specifically  directed  at  the 
tactical  mission  either  from  "objective"  measurement  or  from  subjective  ques- 
tionnaires. It  appears  that  our  understanding  of  cues  required  for  training 
is  at  best  limited,  and  at  worst,  "unquantified." 

In  a recent  article  appearing  in  Astronautics  and  Aeronautics, 
September  1976,  Captain  Terry  L.  Balven,  USAF,  Simulator  SPO  indicates  that 
"We  need  to  know  better  what  cues  aircrews  use  in  performing  their  missions. 

And  we  need  to  know  what  fidelity  these  cues  must  have  for  effective  training." 
In  the  previous  issue  of  the  same  publication.  Dr.  Philip  A.  Reynolds  of 
Calspan  Corporation,  reporting  on  the  results  of  the  recent  AIAA  Visual  and 
Simulation  Conference  in  Dayton,  states  "The  papers  presented  and  the  remarks 
made  by  the  panel  and  the  audience  make  it  clear  that  simulation  has  not 
yet  been  defined  well  enough  for  training  purposes  (military)."  The  panel 
discussion  which  occurred  at  the  meeting  was  also  described  in  the  article 
by  Dr.  Reynolds.  One  result  of  that  discussion  was  that  "The  panel  unani- 
mously urged  the  audience  to  get  to  the  heart  of  the  question:  compare  ground 

with  actual  flight  data."  At  that  same  conference,  Brigadier  General  Benton 
K.  Partin,  USAF,  stated  in  his  address  "We  now  have  several  types  of  six 
degrees  of  freedom  platforms,  g-seats,  and  g-suits  available.  However,  we 
need  to  establish  the  contribution  of  these  motion  cueing  devices  in  the  var- 
ious training  tasks  and  which  device  or  combination  of  devices  can  most  effec- 
tively provide  these  cues.  We  must  establish  the  effects  of  visual  detail, 
realism  and  field-of-view  on  effectiveness  of  training  in  visual  maneuvers 
ranging  from  takeoff  and  landing  to  air-to-air  and  air-to-surface  weapons 
delivery.  Finally,  we  need  to  develop  an  understanding  of  how  all  of  these 
various  cues  interact  in  training  of  aircrew  trainees  and  aircrew  transition- 
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ing  to  and  maintaining  proficiency  in  fighters,  bombers,  or  transports." 

General  Partin  then  went  on  to  indicate  that  AFSC  is  examining  those  areas 
where  there  are  information  voids  in  training  effectiveness  and  he  also  placed 
some  challenges  before  the  R § D community.  The  most  recent  AGARD  Conference 
on  Flight  Simulation,  held  in  The  Netherlands,  included  recommendations 
directed  at  cue  requirements,  i.e.  "proper  generation  of  motion  cues  remains 
a fundamental  problem.". 

In  examination  of  the  literature  for  data  concerning  the  impact 
of  various  cues  on  training,  it  appears  that  there  is  more  data  of  a subjec- 
tive nature  than  objective.  However,  one  fact  remains  clear,  it  is  impossible 
for  the  ground  base  simulator  to  duplicate  the  real  world  situation.  One 
must  take  advantage  of  the  human  perceptual  process  to  create  a desired  and 
cost-effective  illusion  of  reality  sufficient  to  obtain  the  required  level  of 
training.  To  quote  from  the  address  by  Jack  A.  Adams  to  The  Society  of 
Engineering  Psychologists,  (Reference  47)  in  1971,  "I  would  not  consider  the 
money  being  spent  on  flight  simulators  as  staggering  if  we  knew  much  about 
their  training  value,  which  we  do  not.  We  build  flight  simulators  as  realistic 
as  possible,  which  is  consistent  with  the  identical  elements  theory  of  trans- 
fer of  Thorndike,  but  the  approach  is  also  a coverup  for  our  ignorance  about 
transfer  because  in  our. doubts  we  have  made  costly  devices  as  realistic  as 
we  can  in  the  hopes  of  gaining  as  much  transfer  as  we  can."  If  we  examine  the 
panel  discussion  at  the  recent  AIAA  Visual  and  Motion  Simulation  Conference, 
April  1976,  we  find  the  following  comment  offered  by  Dr.  Charles  Hutchinson, 
AFOLR  Life  Sciences  Division:  "I  think  that  it  is  important  that  we  unlearn 

what  we  "know"  about  flight  simulators  because  what  we  "know"  could  prove  to 
be  an  expensive  and  inefficient  hang-up.  If  we  "know"  that  a flying  training 
simulator  has,  or  should  have,  a 6 DOF  motion  system  and  a visual  system  of 
"specifiable"  fidelity,  we  "know"  something  that  is  not  based  on  evidence  pro- 
duced by  studies  of  the  effective  transfer  of  learning  from  training  device 
to  aircraft." 


Now  the  question  remains,  after  this  rather  bleak  introduction 
into  "cues"  as  to  what  do  we  think  we  know  either  experimentally  or  logically 
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about  these  cues.  The  amount  of  data  we  have,  for  example,  on  the  impact  of 
motion  on  training  effectiveness  appears  to  be  contained  in  essentially  two 
reports.  However,  there  are  considerable  data  available  on  attempts  to 
examine  task  performance  with  and  without  motion,  etc.  In  addition,  we  have 
documents  which  at  least  present  to  us  some  subjective  data  on  cue  impact 
and  there  have  been  some  recent  attempts  to  do  a "preliminary"  investigation 
of  various  effects  of  cues.  In  order  to  present  this  data  in  a reasonable 
format,  we  shall  first  separate  the  "cues"  into  motion  related  cues  (e.g. 
moving  platforms,  g-seats) , visual  cues,  and  additional  cues  for  "realism" 
(e.g.  aural). 

At  a recent  Symposium  held  in  April  1976  by  the  Royal  Aeronauti- 
cal Society,  J.  Gundry  presented  essentially  a survey  paper  entitled  "Man 
And  Motion  Cues",  (Reference  48).  This  paper  is  somewhat  similar  in  content 
to  the  same  author's  presentation  at  the  recent  AIAA  Conference,  but  does  pro- 
vide more  extensive  background  data  than  the  latter  paper.  In  this  paper, 
the  author  essentially  attempts  to  catalogue  motion  in  an  aircraft  into  pri- 
marily two  effects: 


• Advantageous  — provide  pilot  with  useful  information 

• Disadvantageous  — motion  sickness,  spatial  disorientation 


The  author  then  proceeds  to  a discussion  of  simulation.  He  introduces  his 
thought  about  present  state  of  knowledge  of  motion  effect  on  pilot  training: 
"Unfortunately  there  is  no  experimental  evidence  that  the  presence  of  flight 
simulator  motion  aids  training. " Of  course,  taken  from  the  opposite  point 
of  view, there  is  no  experimental  evidence  that  appropriate  motion  cues  dis- 
tract from  training.  The  author  then  goes  on  to  illustrate  by  reference  that 
^ "the  presence  of  motion  has  a beneficial  effect  upon  pilot  performance."  He 

then  states  that  this  is  of  little  importance  to  the  training  activity  if 
there  is  no  effect  on  training.  However,  how  does  one  really  measure  the 
effect  and  what  devices  are  applicable  to  conducting  the  required  experiment 
is  not  discussed.  In  addition,  Reference  48  then  summarizes  several  reports 
which  first  of  all  either  indicate  a modification  of  pilot  transfer  function 
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parameters  and/or  control  activity  such  that  in  the  presence  of  motion  these 
characteristics  were  more  representative  of  flight  data  than  when  motion  was 
not  present.  If  one  was  to  apply  a "perceptual"  fidelity  argument,  then  it 
would  appear  that  the  presence  of  motion  does  elicit  a response/stimuli 
situation  more  representative  of  the  real  world  than  when  motion  is  not  present. 
In  addition,  there  are  papers  which  tend  to  indicate  that  as  the  vehicle  dy- 
namics become  less  stable  or  more  unstable,  then  the  presence  of  motion  was 
required  to  maintain  control  by  the  pilot.  In  certain  situations,  control 
by  vision  alone  was  not  possible  (Reference  49).  Thus,  from  these  observa- 
tions, it  is  possible  to  build  a "perceptual"  case  for  motion  in  training, 
most  logically  for  emergency  and  failure  states  at  the  minimum.  With  specific 
regard  to  the  subject  of  engine  failure  training  for  the  KC-135,  Reference  50 
reports  that  "the  pilots  trained  with  a motion  system  performed  significant- 
ly better  than  those  without  a motion  system."  In  addition,  this  reference 
goes  on  to  indicate  a visual/motion  interaction  by  the  statement  "In  almost 
all  cases,  the  improvement  in  training  effectiveness  gained  when  both  the 
visual  and  motion  systems  were  used  was  vastly  superior  to  the  incremental 
gain  achieved  by  using  either  system  alone."  Studies  recently  reported  by 
McLane  and  Wierwille  (Reference  51)  also  indicate  that  for  the  less  complex 
task  of  controlling  an  automobile,  that  motion  (as  well  as  audio)  cues 
had  a significant  influence  on  driver  performance.  Spring,  in  his  AIM 
paper  (Reference  33),  informs  us  that  when  the  simulator  motion  was  turned  off, 
the  Navy  IP's  universally  felt  that  there  was  a significant  reduction  in 
realism,  and  that  there  was  "no  enthusiasm  for  the  simulation  with  this  limi- 
tation." However,  as  the  author  states  "It  should  be  recognized  that  the 
effect  of  motion  on  motivation  and  the  effect  on  transfer  of  learning  are 
not  necessarily  the  same."  On  the  subject  of  motion,  he  states  "The  debate 
as  to  the  quantitative  utility  of  motion  for  air  combat  training  will  con- 
tinue until  hard  evidence  is  obtained."  It  should  be  noted  that  some  recent 
attempts  to  obtain  "hard  evidence"  have  not  put  the  matter  to  rest.  In  a 
recent  paper  entitled  "Platform  Motion  in  Flight  Simulators:  Critical  or 

Nice?",  Dr.  Hagin  of  AFHRL/FT  summarizes  the  results  of  recent  studies  of 
motion/visual  cues  using  ASUPT.  It  is  of  value  to  review  this  article  because 
to  some,  ASUPT  represents  today  "the  pinnacle  in  the  state-of-the-art  in  full 
mission  simulation  capability"  (Reference  1).  The  value  of  the  cited  studies 
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is  questionable  since  the  quality  of  the  motion  and  visual  systems  for  ASUPT 
to  provide  timely  output  is  suspect  (Reference  52).  However,  Reference  53 
does  inform  us  that  these  studies  performed  by  AFHRL/FT  neither  strengthen  nor 
weaken  "the  case  for  platform  motion  as  being  critical  to  the  achievement  of 
substantial  training  transfer."  It  is  also  interesting  to  note  that  in  one 
of  the  cited  experiments,  motion  was  considered  "detrimental"  to  efficient 
simulator  training  and  "that  three  of  the  four  motion  trained  students  exper- 
ienced air  sickness  in  the  aircraft  while  none  of  the  no-motion  trained  stu- 
dents did."  This  may  be  indicative  of  the  quality  of  the  motion  system  for 
ASUPT  rather  than  generalization  of  the  inappropriateness  of  motion  for  train- 
ing. Thus,  concerning  the  value  of  motion  cues,  all  that  can  be  said  is  there 
exists  a significant  void,  however,  one  can  state  that  if  motion  cues  are  pro- 
vided they  must  be  realistic  and  not  produce  false  cues  to  the  pilot.  In 
addition,  motion  systems  should  be  reliable  and  easily  maintained  if  they  are 
to  be  used  properly.  With  regard  to  the  SAAC  motion  system  as  used  during  TAC 
ACES  II,  one  recommendation  is:  "The  SAAC  Motion  System  should  not  be  used  for 

BFM/ACM."  Another  is  a recommendation  to  reconsider  motion  required  in  future 
air-to-air  simulators  because  the  participants  in  TAC  ACES  II  asserted  a de- 
gradation in  training  realism  with  the  presence  of  motion.  Thus,  while  the 
data  available  is  not  sufficient  to  determine  the  cost-effectiveness  of  motion 
in  military  tactical  training,  several  observations  can  be  made: 

• Motion  can  have  a significant  impact  on  aircrew  motivation. 

• Motion  appears  to  provide  a perceptual  environment  closer 
to  the  real  world. 

• No  motion  appears  to  be  better  than  poor  motion  for  training 
applications* 

• Motion  requirements  are  dependent  upon  task  and  vehicle 
dynamics. 
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• Additional  research  on  motion  cues  using  appropriate  and  ade- 
quate facilities  is  required,  with  appropriate  hardware  and 
software  requirements  (time  delays,  washout  circuits). 

In  terms  of  visual  cue  requirements.it  is  obvious  that  for  the 
tactical  military  mission  it  is  essential  to  provide  appropriate  visual  cues. 

The  area  of  concern  is  the  out-the-window  cues  required  for  the  training 
situation  to  represent  the  real  world.  Again, as  with  motion,  there  is  con- 
siderable evidence  that  time  delays  can  have  significant  degrading  impact. 

In  a paper  presented  before  the  AIAA  in  1963,  M.  Aronson  states:  "Studies 

of  the  effect  of  time  lag  between  the  pilot's  control  action  and  the  feedback 
of  the  movement  have  shown  that  it  should  be  in  the  order  of  50-60  milli- 
seconds to  create  the  illusion  that  the  pilot  is  controlling  the  situation." 
Reference  52  indicates  that  the  ASUPT  had  delays  (both  software  and  hardware) 
of  several  times  the  value  cited  above  and  interestingly,  motion  lagged  visual. 

A study  cited  in  Reference  52  indicated  that  with  visual  cues  only,  time 
delay  and  iteration  rates  could  be  the  difference  between  the  ability  of  the 
pilot  to  perform  a formation  flying  task  successfully.  Concerning  the  require- 
ments for  external  visual  cues  in  the  air-to-air  situation,  Reference  54 
states:  "Obviously,  external  visual  cues  are  required,  but  little  is  known 

of  the  specific  cue  requirements  for  air-to-air  combat  or  if  the  interactions 
among  visual  and  other  control  cues  are  important  in  flight  simulation."  This 
paper  then  goes  on  to  describe  an  evaluation  performed  using  a simple  checker- 
board display  to  provide  some  limited  visual  terrain  cues.  The  study  concludes 
that  the  display  evaluated  has  low  subjective  realism  but  high  utility.  It 
should  be  noted  that  no  experimental  data  or  transfer  data  was  presented  in 
the  reference.  A recent  paper  presented  by  Heintzman  and  Shumway  (Reference  55) 
essentially  leads  us  back  to  an  observation  made  by  D.  R.  Tait  in  1970,  that  is, 
total  visual  simulation  of  a complete  flight  is  neither  feasible  nor  practical 
in  the  immediate  future.  Compromises  are  required  based  on  the  device  effec- 
tiveness to  provide  satisfactory  training.  The  most  cost-effective  visual  sys- 
tem may  be  a composite  of  the  various  systems  available.  Concerning  visual 
system  technology.  Reference  55  states  that  the  following  questions  remain  to 
be  answered,  "as  to  what  field  of  view,  brightness,  and  resolution  may  be  re- 
quired for  specific  tasks."  These  questions  are  not  necessarily  new  (References 
56,  57,  and  58).  It  is  obvious  from  the  literature  that  they  have  not  yet  been 
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satisfactorily  addressed  and  as  a result  "objective"  data  is  not  available  to 
specify  cost-effective  visual  systems  for  tactical  combat  training.  Again,  as 
stated  before  with  motion  cues,  the  visual  cues  presented  should  not  be  anta- 
gonistic to  the  user.  For  example,  while  there  is  little  data  to  indicate  the 
impact  of  color  to  visual  simulation  fidelity,  there  does  exist  some  data  to 
indicate  that  color  can  influence  pilot  control  characteristics  (Reference  59) 
and  some  additional  data  to  indicate  that  under  certain  conditions  tJie  addi- 
tion of  color  may  be  mere  distracting  to  the  pilot  than  useful  (Reference  60) . 
Thus,  it  appears  that  again,  a significant  distraction  factor  provided  by  an 
inadequate  cue  representation  will  reduce  the  desired  effectiveness.  At  pre- 
sent, in  the  world  of  visual  simulation,  there  appears  to  be  a significant 
movement  towards  the  use  of  computer-generated  images,  and  the  need  to  increase 
the  number  of  edges,  etc.  Very  little  data  could  be  found  on  to  what  extent 
the  external  view  of  the  world  can  be  simplified.  One  paper  was  uncovered 
which  describes  an  experiment  and  some  preliminary  results  directed  at  elim- 
inating unimportant/unnecessary  real  world  information  to  minimize  CGI  require- 
ments, Reference  61.  The  stated  purpose  of  the  experiment  is  to  establish 
data  to  allow  a reasonable  trade  between  CGI  cost  or  complexity  and  require- 
ments. The  preliminary  results  indicate  that  pilots  do  use  texture  cues  for 
height  estimation  and  that  regardless  of  the  degree  of  "stylization"  there  is 
a greater  tendency  for  height  error  than  distance  error.  Since  a learning 
process  was  observed,  perhaps  it  suggests  that  the  human  visual  experience  has 
normally  been  more  concerned  in  distance  judgement  rather  than  height.  This 
is  somewhat  reinforced  by  the  observation  in  the  cited  paper  that  height  judge- 
ment of  pilots  is  better  than  nonpilots.  Of  course,  this  last  statement  leads 
us  to  another  point,  since  it  is  the  pilot  we  are  attempting  to  understand  in 
the  training  situation,  and  since  differences  have  been  observed  between  the 
performance  of  pilots  and  nonpilots  in  various  simulation  experiments,  research 
into  the  training  effectiveness  of  a flight  simulator,  or  on  the  impact  of  var- 
ious cues  on  the  training  situation  should  use  an  appropriate  subject  as  well 
as  a facility  capable  of  providing  an  applicable  environment. 

To  complete  this  discussion,  let  us  now  spend  some  time  examining 
additional  cues.  These  cues  (e.g.  aural)  can  be  used  to  enhance  the  subjective 
realism  of  a simulation  and  can  also  influence  the  performance  of  the  subject. 
As  discussed  in  Reference  51,  for  example,  the  removal  of  an  aural  cue  provided 
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by  an  engine  noise  generator  reduced  the  subject's  performance  with  respect 
to  velocity  control.  In  addition,  the  aural  cue  tended  to  mask  spurious 
noises  generated  by  the  motion  drive  system.  Again,  there  is  little  data 
available  that  objectively  examines  the  impact  of  these  additional  cues  such 
as  noise,  buffet,  etc.;  however,  as  reported  in  Reference  60,  there  is  subjec- 
tive data  that  both  engine  and  aerodynamic  noise  cues  are  considered  by  a 
majority  of  both  military  and  commercial  pilots  to  be  either  necessary  or 
desirable  for  simulation.  In  a study  reported  by  Starke  and  Wilson  (Reference 
62) , it  was  determined  that  the  simulation  of  buffet  and  vibration  was  con- 
sidered as  "essential  in  the  simulation  of  flight  near  the  edge  of  tne  per- 
formance envelope  ...."  The  authors  continue,  "While  transfer  studies  have 
not  been  accomplished  with  respect  to  fidelity  requirements,  it  is  expected 
that  anything  but  complete  simulation  of  buffet  would  be  pointless  in  prepar- 
ing pilots  to  make  best  use  of  their  aircraft  in  air-to-air  combat."  They 
consider,  aside  from  the  cockpit  motion  system,  that  a system  to  provide  vi- 
bration and  buffet  cues  relative  to  angle  of  attack  is  required  to  provide 
flight  essential  cues. 

If  one  wishes  to  then  generalize  from  the  literature  survey  per- 
formed, only  one  immediate  conclusion  appears  to  be  obvious,  that  is,  there 
is  not  any  data  that  can  substantiate  from  objective  training  effectiveness 
consideration,  the  necessary  or  the  desirable  cues,  the  appropriate  cue  inter- 
action, etc.  required  to  provide  appropriate  fidelity  for  the  training  situa- 
tion. It  is  also  somewhat  obvious  that  many  existing  simulation  facilities 
such  as  SAAC  and  ASUPT  may  be  presently  inadequate  for  use  as  a facility  to 
examine  the  effectiveness  or  fidelity  requirements  on  many  cues.  It  is  also 
obvious  from  a review  of  the  existing  data,  that  a device  promised  to  provide 
high  fidelity  simulation  is  of  little  value  if  it  introduces  significant 
spurious/distracting  cues  or  is  not  reasonably  reliable  or  cannot  be  easily 
maintained.  These  last  two  items  will  have  a significant  impact  on  any  cost- 
effectiveness  study  that  requires  abnormally  high  utilization  rates  for  a com- 
plex device  in  order  to  justify  its  procurement.  It  also  appears  that  it 
may  be  better  to  eliminate  a cue  in  a training  situation  than  to  provide  what 
is  considered  to  be  a "false"  cue. 
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Simulator  Data  Requirements/Acceptance  Evaluation 


During  the  literature  review,  it  became  obvious  that  there  was 
much  subjective  data  which  indicated  a significant  inadequacy  of  existing 
simulators.  While  many  pilots  fel*  that  simulators  could  be  used  for  proce- 
dural training  and  emergency  training,  the  general  handling  qualities  area  was 
considered  to  be  inadequate.  Thus,  it  appeared  to  be  of  value  to  survey  the 
literature  for  some  definitive  reports  on  simulator  data  requirements  and  accep- 
tance evaluation  procedures.  It  is  obvious  that  the  full  potential  of  the 
flight  simulator  cannot  be  approached  without  a reasonable  ability  on  the  part 
of  the  device  to  present  to  the  pilot  the  flying  qualities  and  performance 
characteristics  of  the  actual  aircraft.  In  other  words,  to  instill  confidence 
in  the  subject  that  the  time  spent  in  the  simulator  is  being  used  effectively, 
the  program  should  be  ’’credible",  that  is,  the  ultimate  user  must  be  reasonably 
sure,  at  least  perceptually,  that  the  simulator  "flies  like  the  aircraft." 

The  remainder  of  this  discussion  will  present  observations  made 
from  the  literature  survey  on  aerodynamic  data  package  requirements  and  also 
observations  made  on  the  topic  of  simulator  fidelity  assessment  directed  toward 
acceptance  testing.  These  two  subjects  are  definitely  interrelated  since  it 
appears  that  one  of  the  great  shortcomings  of  existing  simulation  equipment  is 
to  produce  a representative  flying  qualities  environment.  Even  for  the  case  of 
simulators  used  for  training  in  the  Airline  environment,  the  following  opinion 
was  expressed  in  1974  (Reference  31)  by  D.  P.  Davies:  "Flight  Simulation  has 

made  good  steady  progress  over  the  years  and  is  now  being  given  significant 
credit  for  its  ability  to  reproduce  aircraft  flying  qualities  in  a device  on 
the  ground.  However,  the  simulator  world  is  tending  to  claim  too  much  credit 
for  its  present  level  of  achievement."  He  then  goes  on  to  give  advice  to  the 
various  parties  involved  in  a simulation  program,  which  in  essence  provides 
the  framework  for  an  acceptance  program: 
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• Advice  to  airframe  manufacturers 

• supply  an  accurate  data  package 

• update  data 

• provide  pilots  to  simulator  manufacturers  and  users 

• Advice  to  the  simulator  manufacturer 

• insist  on  an  up-to-date  data  package  from  the  airframe 
manufacturer 

• develop  a current  test  pilot  capability 

• Advice  to  the  simulator  operator  to  achieve  a high  quality 
simulator 

• do  not  "twiddle"  to  achieve  good  handling  qualities 

• use  the  best  assistance  from  simulator  manufacturers 
and  airframe  manufacturers 

• develop  your  own  competent  staff 

• insist  on  an  accurate  up-to-date  data  package 

• Advice  to  the  simulator  operator  on  use  of  the  simulator 

• essentially  maintain  the  realism  of  actual  flight,  do 
not  use  training  features  in  an  inappropriate  manner  or 
permit  abuses  because  the  "simulator  is  a simulator  and 
not  an  airplane" 

Thus,  it  appears  that  the  initial  and  possibly  most  fundamental  concern,  and 
logically  so,  is  the  accuracy  of  the  data  package.  According  to  D.  A.  Clutz, 
Reference  63,  those  portions  of  the  data  package  most  poorly  defined,  in 
general,  are  the  aerodynamic  coefficients,  the  control  coefficients  and  the 
cockpit  and  surface  control  characteristics.  It  is  well  known  that  this  data 
is  critical  to  the  design  and  use  of  an  aircraft  so  it  becomes  essentially  a 


logical  extension  to  believe  these  parameters  should  also  be  considered  critical 
to  the  flight  simulator.  The  questions  which  remain  are  more  related  to  who  is 
responsible  to  supply  the  data,  what  level  of  complexity  is  required  in  the 
data,  and  in  addition,  what  is  the  impact  of  the  real  time  digital  computational 
effects  or  the  impact  of  other  system  dynamics  on  the  handling  qualities/per- 
formance fidelity  achieved  by  the  training  device.  This  last  question  is  ob- 
viously not  as  simple  as  just  implementing  the  data  in  real  time,  in  his  arti- 
cle recently  published  on  the  U.S.  Army  new  approach  to  this  problem  area, 

Lt/Col  Robert  L.  Catron  reminds  us  of  the  following  fact: 

• definitive  data  which  describes  the  aircraft  handling  qualities/ 
characteristics  for  all  required  flight  conditions,  etc. 

is  simply  not  often  available. 

He  then  also  reminds  us  of  the  reason  why  this  is  more  often  the 
situation  than  not: 

(1)  developing  a precise  math  model  is  not  a primary  goal 
of  the  airframe  manufacturers 

(2)  there  are  often  "holes"  even  in  the  most  rigorous  flight 
test  programs 

(3)  flight  test  can  be  made  obsolete  as  the  aircraft  evolves 
through  the  development  phase 

(4)  extensive  flight  testing  to  provide  an  accurate  data  package 
update  can  be  quite  expensive. 

The  articles  then  go  on  to  describe  the  new  approach,  based  on  establishing  a 
qualified,  experienced  evaluation  team.  However,  the  use  of  an  evaluation  team 
can  still  involve  several  problems,  relating  to  the  "accuracy  of  the  pilots 
perception".  Reference  63: 

• relationship  of  pilots  aircraft  to  the  average  aircraft 
of  the  type  required  for  the  training  device 

• pilot  currency  and  proficiency  in  the  test  being  conducted 

• pilot  adaptation  to  the  simulator. 

On  this  last  issue,  various  estimates  indicate  after  about  4 to  8 hours  on  the 
simulator,  the  pilot  mav  no  longer  be  able  to  provide  valid  judgments  on 
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discrepancies  between  the  handling  qualities/performance  of  simulator  and  the 
aircraft  (References  63  and  64).  In  addition,  these  judgments  are  subjective 
and  may  require  extensive  coordination  by  a team  of  pilots,  engineering,  flight 
test,  computer  and  simulation  experts  to  resolve  problems  into  a valid  hard- 
ware or  software  fix.  An  approach,  to  attempt  to  quantitize  these  subjective 
evaluations  into  a form  that  can  be  more  readily  used  in  engineering  analysis 
has  been  suggested  in  Reference  25.  Nevertheless,  it  appears  that  there  are 
essentially  two  fundamental  issues,  not  so  much  that  good  data  is  required 
but  more  so  who  should  be  responsible  for  providing  the  data,  and  then  what 
is  the  most  effective  and  efficient  concept  to  apply  to  acceptance  testing. 

On  the  subject  of  the  validity  and  adequacy  of  the  data,  we  should 
remember  that  normally  during  the  design  of  an  aircraft,  an  initial  data  package 
is  developed  by  the  airframe  manufacturer  for  use  in  an  engineering  simulator. 
While  this  data  package  may  not  be  as  complete  as  the  simulator  manufacturer 
would  desire  (for  certain  obvious  reasons)  it  is  the  best  guess  for  the  initial 
design/handling  quality  studies  performed  by  the  airframe  prime  (provided  of 
course  he  is  obligated  to  perform  a simulation).  The  flight  testing  of  the 
aircraft  itself  against  its  specification  will  supply  significant  data  if  pro- 
perly obtained,  either  by  classical  analysis  or  by  more  recent  computer  systems 
developed  for  flight  test.  Parameter  identification  techniques  can  be  utilized, 
to  also  provide  additional  data,  or  at  least  evaluate  the  adequacy  of  the 
existing  data  package.  For  an  aircraft  in  operational  use,  another  source  of 
data  are  the  using  command  pilots  and  aircrews.  However,  complaints  must  be 
transferred  into  viable  fixes  to  the  training  device,  and  there  is  some  indica- 
tion in  the  literature  that  the  simulator  manufacturers  have  been  ineffective 
in  this  task,  (Reference  64),  possibly  as  a result  of  a breakdown  in  communi- 
cations between  simulator  specialists  and  operational  pilots.  References  52 
and  64  also  indicate  that  increasing  the  capability  of  a training  device  (e.g. 
adding  an  external  visual  flight  system)  can  magnify  existing  simulator  dis- 
crepancies, and  training  value  could  subsequently  suffer  without  proper  system 
integration  or  modification.  For  example,  the  relationship  between  the  visual 
and  motion  time  delay  for  a given  device  could  influence  the  ability  of  the 
pilot  to  detect  the  motion  delay.  Reference  52. 
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Reference  64,  presents  the  program  team  concept  used  by  the  U.S. 

Navy  in  an  attempt  to  improve  the  fidelity  of  a flight  training  device.  Essen- 
tially this  approach  applied  a team  concept,  involving  personnel  with  various 
capabilities  and  backgrounds,  doing  their  thing  and  interacting  to  identify  and 
quantify  device  fidelity  problems.  The  approach  used  to  validate  the  simulator 
involved  the  use  of  performance  and  stability  and  control  tests  essentially 
equivalent  to  those  required  of  the  aircraft.  The  procedure  involved  an  iterative 
process,  examining  the  fidelity  at  various  levels  of  device  complexity,  e.g., 
fixed  base  first,  then  add  motion  system  and  re-evaluate,  etc.  This  approach 
appears  to  be  reasonable  and  successful,  but  it  also  indicates  that  perhaps  the 
device  was  initially  accepted  with  insufficient  testing.  That  is,  the  modeling 
problems  were  always  there  in  the  data  package.  There  are,  however,  no  discus- 
sions presented  on  the  impact  of  the  improper  model  data  on  any  students  trained 
in  the  device  prior  to  the  fidelity  improvement  exercise. 

In  summary,  it  appears  that  a properly  coordinated  team  effort, 
with  designated  responsibility  is  essential  to  both  obtaining  a valid  data 
base  for  the  model  and  to  provide  acceptance  testing.  It  is  deemed  essentially 
that  the  government  personnel  charged  with  airframe  system  development  respon- 
sibility and  their  counterparts  charged  with  simulator  system  development 
interact  in  controlled  manner  to  insure  a proper  data  base  is  provided.  Flight 
testing  will  most  definitely  have  to  be  performed  and  data  well  documented 
especially  at  the  edge-of-the-envelopc  if  the  training  to  be  provided  on  a 
simulator  at  those  conditions  is  expected  to  be  meaningful.  It  becomes  highly 
possible  for  negative  transfer  to  occur  at  these  critical  conditions,  thereby 
increasing  the  potential  risk  on  "first-shot"  transfer  to  the  aircraft.  A 
reasonable  set  of  requirements  to  obtain  an  appropriate  data  base,  designate 
responsibility  and  establish  the  framework  for  development  of  acceptance  testing 
procedures  can  be  obtained  by  following  the  advice  of  D.P.  Davies' (Reference  31) 
augmented  by  the  Simulation  Criteria  Enhancement  approach  advociated  by 
D.A.  Clutz  in  Reference  63.  The  recent  experience  obtained  by  the  U.S.  Army 
and  U.S.  Navy  in  their  acceptance  testing  the  fidelity  improvement  programs 
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provides  valuable  insight  into  how  to  develop  team  testing  procedures  that  are 
required  to  insure  a reasonable  level  of  credibility  in  a training  device. 

Pilot  Survey  on  Simulation 

As  has  been  stated  many  times  in  this  report,  it  was  very  difficult, 
if  not  impossible,  to  obtain  objective  evidence  of  the  value  of  simulators  for 
training  the  complex  skills  required  by  pilots  especially  for  a tactical  situa- 
tion. Most  of  the  data  obtained  in  the  literature  survey,  seemed  to  be  judgments 
made  by  government/military  personnel  committed  (for  whatever  reasons)  to  simu- 
lation, the  "expert"  opinions  of  engineering  psychologists  and  claims  for  de- 
vices made  by  people  with  a vested  interest  in  the  future  simulation  procurement 
(manufacturers,  training  organizations,  etc.).  Thus  it  was  decided,  if  one  is 
to  search  for  opinion,  then  search  for  the  opinion  of  the  ultimate  user,  the 
pilot.  Specific  surveys  have  been  recently  obtained  in  the  TAC  ACES  program 
(this  will  be  the  subject  of  a subsequent  section  in  this  report),  the  purpose 
of  this  section  is  to  report  on  a more  general  survey  obtained  on  the  attitude 
and  opinions  expressed  by  both  military  and  commercial  pilots  on  the  various 
facets  of  flight  simulators  used  as  training  devices  (Reference  60).  This 
reference  reports  on  a study  conducted  by  the  RAF  using  military  pilots  and 
the  British  Airline  Pilots'  Association  using  Airline  Pilots  to  obtain  user 
opinions  on: 

• problem  areas 

• system  benefits 

• future  system  goals 

• establishing  priorities 

It  should  be  noted,  that  if  a similar  general  survey  has  been  conducted  by 
U.S.  Military  Organizations  or  Airlines  Pilots'  Association  it  was  not  uncovered 
during  our  literature  survey.  The  importance  of  these  surveys,  should  not  be 
minimized  since  they  can  be  very  useful  for  determination  of  those  facets 
of  a training  device  that  would  increase  the  credibility  of  the  device  in  the 
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mind  of  the  ultimate  user.  The  data  to  be  presented  in  the  remainder  of  this 
discussion  is  extracted  from  Reference  60.  The  reference  is  separated  into 
first  the  RAF  data  and  secondly  the  Airline  Pilots  data.  This  separation  will 
be  maintained  for  the  convenience  of  any  reader  who  desires  to  cross  check 
with  the  original  data  presented  in  the  cited  reference. 

RAF  DATA 

• Motion  Simulation  - Assessment  of  roll  and  pitch  simulation 
indicated  that  the  majority  of  pilots  surveyed  considered  the  existing  systems 
as  "quite  realistic"  however  approximately  30%  of  the  pilots  indicated  that 
the  systems  were  considered  as  unrealistic. 

• Motion  System  Requirements  - These  requirements  appear  to  be 
task/type  aircraft  dependent.  Examination  of  the  data  for  the  tactical  situation 
indicates  that  a majority  of  pilots  surveyed  consider  roll,  pitch,  turbulence 
and  vibration  as  either  "necessary"  or  desirable.  The  sway  degree  of  freedom 
was  either  not  evaluated  or  not  recommended,  the  table  in  the  reference  is  not 
clear  on  this  subject.  Yaw  degree  of  freedom  had  split  opinion  (essentially 

50%  returns  considered  as  desirable,  and  50%  as  undesirable)  for  the  interceptor 
task,  but  definitely  considered  necessary  for  the  low  level  attack  task.  The 
surge  and  heave  degrees  of  freedom  were  essentially  split  for  the  low  level  task, 
however,  the  majority  of  data  appears  to  indicate  a desirability  for  these 
degrees  of  freedom.  While  this  data  indicates  some  pilot  priorities  on  motion 
requirements,  as  the  authors  state  " clearly  some  rigorous  work  is  still  required 
to  identify  those  motions  which  enhance  training,  those  which  do  not  affect 
training  but  do  influence  the  trainee's  attitudes  to  the  simulator,  and  those 
which  can  be  safely  dispensed  with.  Some  similar  work  on  pilots  attitudes  on 
required  motion  is  presented  by  Starke  and  Wilson  in  Reference  62. 

• Visual  Simulation  - assessments  of  existing  visual  flight  attach- 
ments for  ground  taxi,  take-off  and  landing  flight  phases  indicate  that  while 
many  thought  that  the  attachments  were  quite  realistic  for  these  phases,  again 
almost  30%  or  more  consider  the  fidelity  as  unrealistic. 
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• Visual  Simulation  Requirements  - in  all  cases,  pilots  indicated 
the  outstanding  requirement  for  visual  simulation  was  for  take-off  and  landing 
phases.  For  the  low  level  attack  requirements,  the  mission  phase  and  examina- 
tion of  aircraft  handling  in  the  air  the  survey  data  indicated  a preference  for 
visual  simulation  capability.  Interceptor  pilots  indicated  the  predominant  re- 
quirement for  visual  simulation  capability  was  essentially  for  take-off  and  landing. 

• Required  Visual  Systems  Improvements  - the  vast  majority  of  the 
military  pilots  indicated  that  greater  field-of-view  was  the  single  most 
important  required  improvement  for  visual  simulation.  This  was  followed  by 
collimation  and  greater  depth  of  focus,  while  the  next  priority  went  to  in- 
creased resolution.  Information  of  this  type  is  valuable  in  establishing 
research  objectives. 

• Auditory  Simulation  - the  significant  majority  of  pilots,  es- 
pecially for  the  low  level  attack  mission  consider  engine  noise  and  aerodynamic 
noise  as  necessary  and  desirable  for  simulation.  While  the  data  indicates  that 
the  present  capability  for  engine  noise  simulation  was  considered  realistically 
done,  the  data  indicates  that  improvement  is  required  for  aerodynamic  noise, 
which  was  considered  to  be  "not  very  realistic",  at  best,  by  the  majority  of 
respondents. 

• Handling  Qualities  Simulation  - the  stated  aircrew  requirements 
for  maximum  fidelity  in  this  area  appears  to  be  unsatisfied.  The  majority  of 
pilot  responses  indicate  that  the  existing  simulation  capability  in  this  area 
was  at  best  unrealistic. 

• Training  Value  of  Simulators  - in  general,  the  authors  indicate 
that  it  can  be  supposed  that  the  aircrew  were  relatively  satisfied  with  simulator 
training.  In  the  question  of  simulator  time  requirements  in  conversion,  the 
majority  of  pilots  responding  to  this  question  saw  "no  requirement  for  changing 
the  time  allotted."  Concerning  time  distribution  for  other  training  areas 

(e.g.  mission  execution  and  general  handling,  most  respondents  felt  unqualified 
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to  address  that  topic.  While  70%  thought  that  the  use  of  the  simulator  in 
training  was  successful,  the  simulator  was  rated  highest  in  training  emergencies. 
It  was  considered  less  effective  relative  to  the  aircraft  for  mission  training 
and  essentially  considerably  less  useful  for  general  handling.  Of  the  pilots 
surveyed,  45%  or  more  agreed  that  the  simulator  checks  were  adequate  for  pre- 
dicting performance  in  the  aircraft,  however  50%  of  all  respondents  were  not 
in  favor  of  "objective"  (computer  determined)  performance  assessment.  This 
could  indicate  that  pilots  prefer  to  be  taught  and  evaluated  by  their  peer 
group  according  to  the  authors.  A survey  conclusion  is  that  "the  usefulness  of 
the  simulator  for  other  than  procedual  work  and  emergencies  was  thought  to  be 
limited." 


• Training  Instructor  Requirements  - in  excess  of  90%  of  the  air- 
crews surveyed  thought  that  the  instructor  should  be  current  on  the  aircraft 
being  simulated,  and  that  teaching  ability  was  required.  Instruction  in  teaching 
techniques  was  considered  to  be  either  necessary  or  desirable  by  99%  of  the  air- 
crews responding  to  the  questionnaire.  The  reference  states  that  attitude  of 
the  trainees  towards  the  simulator  for  training  can  be  significantly  influenced 
by  the  role  and  status  of  the  instructor. 

• Conclusions  Based  on  RAF  Survey  - 

• Although  current  motion  systems  appear  to  be  considered 
reasonable,  improvements  in  the  quality  of  simulation 
may  be  desirable. 

• Visual  systems  need  improvements,  most  likely  in  field- 
of-view,  depth-of-focus , and  image  collimation 

• Simulation  of  engine  and  aerodynamic  noise  is 
considered  highly  desirable. 

• Simulation  for  other  than  procedual  work  and  emergencies 
was  considered  limited,  however,  it  was  considered 
satisfactory  at  least  for  these  two  cited  training  areas. 


The  remainder  of  this  discussion  will  present  a summary  of  the  data 
obtained  from  the  Airline  Pilots  responses  to  a questionnaire  on  the  value  of 
simulators  for  training  and  checking  pilot  performance.  The  following  list 
indicates  the  most  significant  criticisms  of  the  various  simulation  facilities 
in  terms  of  descending  order: 

• Poor  Simulation  of  Aircraft  Handling  Characteristics 

• Poor  Motion  Simulation 

• Motivational  Problems 

• Poor  Simulation/Representation  of  Cockpit,  Controls,  etc. 

• Simulator  Reliability 

The  survey  indicated  that  opinion  was  essentially  split, on  pilot/ 
aircrew  satisfaction  with  the  refresher  training/competency  checks  and  instru- 
ment ratings  renewals  as  performed  in  the  simulator,  between  "satisfied"  and 
"not  satisfied."  Again,  as  in  the  RAF  study,  the  majority  thought  that  pro- 
cedures and  emergencies  were  trainable  in  the  simulator,  while  handling  practice 
could  not  be  performed.  Concerning  motion,  it  appears  that  pitch  followed  by 
yaw  and  roll  were  considered  as  motion  requirements  for  simulation.  As  with  the 
RAF  data,  the  majority  of  Airline  Pilots  thought  that  engine  noise,  aerodynamic 
noise,  etc.  were  either  necessary  or  desirable  for  simulation.  The  existing 
simulations  appear  to  be  adequate  in  representing  engine  noise,  however,  improve- 
ment is  required  to  adequately  simulate  aerodynamic  noise.  With  respect  to  the 
value  and  adequacy  of  existing  visual  flight  attachments,  the  ratings  appear  to 
be  strongly  correctable  with  the  training  device,  however,  at  best  they  were 
considered  as  useful  for  some  maneuvers.  Many  responses  indicated  that  the 
visual  simulation  was  inadequate/distracting  especially  with  color.  In  general, 
there  appears  to  be  a general  approval  of  the  training  presently  obtained  in 
the  simulator,  especially  to  obtain  more  practice  without  demonstration  of 
competency.  In  summary,  these  results  tend  to  generally  reinforce  the  results 
obtained  from  the  RAF  study.  Again,  while  these  results  are  obviously  subjective, 
they  demonstrate  some  short  term  goals  that  are  considered  by  the  trainee  to  be 
desirable  and  would  tend  to  increase  the  'credibility'  of  flight  simulator 
training.  The  pilot  appears  to  be  willing  to  accept  this  training,  at  least  for 
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procedural  tasks  and  emergencies.  The  cited  paper  concludes  on  the  importance 
of  achieving  greater  serviceability  and  consistency  from  training  devices, 
that  required  utilization  would  interfere  with  the  desire  to  provide  pilots 
with  "free  time"  for. training  in  the  simulator,  and  that  it  may  not  be  econ- 
omic to  build  devices  with  the  complexity  required  to  achieve  "zero  time". 

The  results  of  the  cited  pilot  surveys  indicate  that  using  simu- 
lation devices  for  standard  procedural  and  emergency  procedures  training  would 
be  acceptable  to  experienced  military  pilots.  Our  previous  discussion  indi- 
cated that  it  was  also  procedural  skills  that  were  most  easily  forgotten. 

Thus  simulator  training  could  prove  to  be  the  most  efficient  method  for  teach- 
ing and  maintaining  pilot  proficiency  of  procedural  skills. 

Full  Mission  Simulation  (A  Case  History) 

It  becomes  apparent  from  a review  of  the  literature  that  there 
seems  to  have  been  placed  in  motion  a military  commitment  to  obtain  full  mission 
simulation  devices.  This  appears  to  have  been  brought  about  by  several  factors, 
but  primarily  motivated  by  the  desire  that  reduction  of  flight  hours  cannot 
reduce  the  combat  readiness  requirements.  In  additiion,  the  argument  for 
training  in  simulators  to  reduce  risk  while  examining  tasks  in  a hostile  envi- 
ronment (i.e.,  the  impossibility  criterion)  has  also  been  cited  as  a justifica- 
tion for  needing  full  mission  simulation  capability,  although  it  is  also  recog- 
nized that  the  simulator,  no  matter  "how  realistic  it  may  be  in  the  military  ap- 
plication, never  can  nor  will  replace  live  flying  experience,  " Reference  39.  Thus, 
while  there  appears  to  be  strong  justification  for  such  a device,  the  question 
is  can  the  full  mission  simulator  be  delivered  and  is  it  "cost-effective"  to 
develop  such  a training  system.  A search  was  performed  to  attempt  to  examine 
the  full  mission  simulator.  The  only  fairly  well  documented  background  on 
this  level  of  capability  appears  to  be  related  with  the  RAF  experience  concern- 
ing the  Phantom  Full  Mission  Simulator.  It  was  deemed  to  be  of  value  to  this 
present  study  to  summarize  this  particular  case  history  in  a chronological 
manner. 
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This  case  history  essentially  starts  with  an  article  presented  in 
1970  Royal  Aeronautical  Symposium  (Reference  65),  primarily  concerning  concept, 
proceeds  to  an  article  presented  in  1973  (Reference  66)  concerning  what  was 
achieved  and  suggesting  an  interim  solution  to  the  deficiency  of  the  device 
and  is  presently  concluded  by  a most  recent  presentation  at  the  1975  AGARD 
Conference  on  Flight  Simulation.  This  case  history  documents  a situation 
where  based  on  a lack  of  background  expertise,  the  "inspired  guesswork"  did 
not  produce  the  desired  capability.  As  will  be  indicated  by  the  following 
presentation,  the  gap  between  the  best  laid  plans  and  the  realities  of  simu- 
lation capability  have  not  yet  produced  the  desired  training  potential  for  the 
simulation  facility  under  discussion. 

Reference  65  sets  the  stage,  as  it  provides  to  us  the  background 
discussion  as  to  the  need  for  full  mission  simulation,  the  general  hardware 
requirements,  the  obvious  advantages  (aside  from  financial  and  flight  safety), 
how  the  device  will  be  utilized  and  the  commitment  to  the  device,  especially 
considering  that  they  "do  not  provide  an  end  in  themselves  but  rather,  by  sup- 
plementing the  decreasing  flying  hours  available  from  our  limited  resources, 
they  help  to  keep  the  point  of  the  services  needle  sharp."  Our  next  document, 
presented  in  1973  (Reference  66),  states:  "Training  results  and  opinion  with- 

in the  Service  have  shown  that  our  significant  commitment  to  the  use  of  sophis- 
ticated full  mission  flight  simulators  has  yet  to  produce  the  degree  of  produc- 
tive training  that  was  initially  envisaged."  The  limitations  cited,  based  on 
operational  experience  for  this  lack  of  achievement,  are  the  following: 

• Servicing  problems 

• Reliability  and  maintainability  of  the  systems 

• Limitations  in  the  visual  systems 

• Restrictions  in  gaming  area 

• Lack  of  qualified  simulator  instructors 

• Difficulties  with  building  facilities 

• Service  contention  with  respect  to  the  value  of  motion  and 
visual  systems 


This  reference  goes  on  to  tell  us  that  the  Air  Staff,  based  on  the  inability 
of  the  full  mission  simulator  to  "effect  the  improvements  and  savings  in  fly- 
ing hours,"  instituted  a task  to  quantify  "the  value  of  flight  simulation  to 
the  military  training  task."  The  reference  informs  us  that  the  Air  Staff 
required  this  to  be  achieved  "by  scientific  methods."  Obviously,  the  time 
for  guesswork  was  coming  to  abrupt  finish  and  a long  term  commitment  had  been 
formulated  to  investigate  this  task.  However,  an  "Interim  Policy"  was  also 
formulated.  This  was  to  first  of  all  progressively  relegate  to  the  simulator 
all  instrument  training  and  secondly,  to  evaluate  what  areas  which  do  not 
receive  adequate  training  in  flight  could  be  supplemented  by  the  use  of  the 
simulator.  The  long  term  policy  would  be  to  use  streaming  trials  to  evaluate 
and  quantify  "the  value  of  various  device  capabilities."  The  author  states 
"We  believe  that  our  withdrawal  from  the  full  mission  concept  is  both  realistic 
and  rational;  future  simulators  will  be  tailored  to  meet  the  specific  require- 
ments of  the  particular  training  task."  This  appears  to  be  a commitment  to 
part-task  training  devices.  To  conclude  this  case  history,  at  least  for  the 
present,  a recent  report  by  a member  of  the  Ministry  of  Defense  was  presented 
at  the  AGARD  Conference  held  in  The  Netherlands  in  1975  (Reference  26).  In 
essence,  this  article  informs  us  that  the  "Long  Term"  research  policy  toward 
quantification  of  the  value  of  this  full  mission  simulator  in  terms  of: 

• Flight  hours  that  can  be  substituted 

• Device  capability/complexity 

is  on-going.  Thus,  the  "hard"  data  is  not  in  yet  and  the  true  value  of  full 
mission  simulation  still  remains  quantitatively  undefined.  Thus,  to  review 
the  facts  of  this  case: 

• Subjective  assessment  was  used  to  justify  procurement  of  de- 
vices to  achieve  desired  benefits 

• Subsequent  operational  experience  did  not  indicate  the  desired 
goals  were  being  achieved 
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• The  high  cost  and  complexity  of  these  devices  has  prompted  a 
study  to  determine  and  quantify  the  training  effectiveness  of 
full  mission  simulation 

• The  "answers"  are  not  yet  available 

Thus,  from  this  case  history,  it  appears  to  be  highly  desirable  to  review  any 
planned  procurement  of  full  mission  simulation  capability  and  determine  whether 
or  not  dedicated  part-task  trainers  could  not  prove  a more  viable  and  presently 
more  "cost-effective"  solution  to  training  requirements.  In  the  opinion  of 
the  authors,  based  on  the  literature  survey,  the  case  for  the  full  mission 
simulation  cannot  be  based  on  the  demonstrated  practicality  of  such  a complex 
training  device  although  such  capability  would  appear  to  be  both  desirable 
and  potentially  technically  feasible.  While  a need  may  exist,  the  present 
state-of-the-art  of  simulation  technology  does  not  appear  to  be  able  to  sup- 
port a "cost-effective"  tactical  full  mission  simulator.  As  recently  con- 
cluded by  Gum  and  Swab  in  Reference  16,  "The  state-of-the-art  is  adequate  in 
many  areas  to  support  the  acquisition  of  effective  training  devices  which 
will  allow  considerable  aircraft  flight  training  time  to  be  saved.  Much  de- 
velopment remains  to  be  done  in  order  to  satisfy  all  of  the  using  command 
needs  and  to  permit  training  simulators  to  achieve  their  full  potential." 

TAC  ACES  I and  II 


As  part  of  the  data  base  for  the  literature  survey,  two  documents 
were  furnished  to  us  concerning  the  TAC  ACES  program  (References  30  and  34) . 
These  programs  are  based  on  a TAC  established  requirement  to  supplement  exist- 
ing F-4  air-to-air  combat  flight  training  using  existing  state-of-the-art 
simulation  facilities.  The  TAC  ACES  I program  was  performed  on  an  industrial 
simulator  (VACS),  while  the  TAC  ACES  II  program  was  performed  on  the  SAAC  simu- 
lator. The  purpose  of  this  section  is  to  review  the  conclusions  and  recommen- 
dations of  these  documents  as  they  impact  on  current  state-of-the-art  simulator 
technology  to  provide  adequate  "supplemental"  training.  It  should  be  remem- 
bered that  there  have  not  been  any  streaming  trials  conducted  to  evaluate  the 
transfer  effectiveness  of  the  training  received  by  the  pilot  participating  in 
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the  TAG  ACES  programs.  Any  evaluation  performed  of  the  effect  of  the  simulator 
training  on  air-to-air  combat  proficiency  is  based  on  improvements  in  score  in 
the  device  (wnich  could  be  nothing  more  than  a pinball  machine  effect)  and 
subjective  statements  made  by  the  pilots  who  participated  in  the  program.  It 
should  be  stated  at  the  onset  of  this  review  that  both  documents  indicate 
that  there  was  sufficient  lack  of  realism  that  could  result  in  "development 
of  bad  habits"  and  "negative  training"  unless  the  courses  are  conducted  as  a 
highly  controlled,  short  term  training  program.  For  the  programs  conducted, 
these  devices  were  operated  as  dedicated  part- task  training  devices.  Signi- 
ficant deficiencies  were  noted  and  some  improvements  indicated  as  required  to 
continue  the  training  program  concept,  although  on  both  programs  a majority 
of  subjects  considered  the  value  of  the  training  received  (even  with  the 
fidelity  limitations)  as  either  good  or  excellent.  This  suggests  that  some 
dedicated  training  program,  with  the  ability  to  relate  academics  quickly  with 
a facsimile  of  the  operational  environment  can  be  effectively  used  to  supple- 
ment aircraft  training  even  with  reduced  fidelity.  The  results  of  the  programs 
do  not  justify  any  extrapolation  that  the  training  provided  could  be  used  as 
a substitute  for  actual  simulated  air-to-air  combat  training  in  the  aircraft. 

It  is  significant  to  note  also  that  one  of  the  better  uses  of  the  simulator 
was  considered  to  be  "switchology,"  indicating  that  possibly  the  most  effec- 
tive training  that  can  be  presently  provided  by  these  devices  is  in  proce- 
dural skills  required  for  air-to-air  combat.  The  results  of  these  programs 
indicate  that  full  mission  simulation  capability  for  the  tactical  situation 
is  not  presently  state-of-the-art,  however,  with  correction  of  certain  defi- 
ciencies supplementary  training  with  dedicated  part-task  devices  is  achiev- 
able in  the  short  term,  but  requires  a highly  controlled  training  environment. 
The  following  list  indicates  the  significant  simulation  deficiencies  identi- 
fied from  pilot  comments  on  the  TAC  ACES  I program: 

• Target  aircraft  imagery  is  inadequate 

• Cockpit  control  loading  is  unrealistic 

• Fidelity  of  aircraft  handling  qualities  and  performance  is 
deficient,  especially  at  high  AOA 

• Weapons  scoring  was  inadequate 

• AIM-9  seeker  tone  unrealistic 
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• Prerecorded  maneuvering  target  is  of  limited  value 

• Visual  altitude  cues  were  inadequate 

• Engagement  scene  display  format 

• Target  presentations  were  unsuitable  for  formation  flying 

• "g"  indications  ineffective,  tendency  to  use  excessive  g, 
could  lead  to  bad  habit 

• Program  limited  to  1 on  1,  cannot  explore  operational  tactics 

• Simulator  layout  restricts  aft  visibility  and  use  for  training 
in  DCM 

Even  with  these  limitations,  the  overall  opinion  of  the  pilots  participating 
in  the  TAC  ACES  1 program  considered  it  to  be  valuable  training  especially  for 
an  inexperienced  ACM  pilot.  Caution  is  indicated  due  to  device  limitations 
and  the  possibility  of  developing  bad  habits  because  of  the  lack  of  realism. 

The  comments  indicate  that  most  pilots  considered  it  as  a good  ACM/BFM  pro- 
cedural trainer  and  a complement  to  actual  flight.  Perhaps  some  of  the  most 
revealing  comments  on  the  value  of  the  TAC  ACES  I training  program  were  the  fol- 
lowing: "Any  realistic  training  is  better  than  what  we  currently  have,  which 

is  nothing."  "The  simulator  would  be  of  least  benefit  to  highly  experienced  pilots 
because  it  would  make  very  little  addition  to  their  state  of  proficiency  or 
learning.  The  simulator  would  not  provide  the  finer  points  of  air  combat  that 
could  only  be  achieved  in  an  actual  situation  (real  aircraft)."  The  primary 
recommendations  presented  with  respect  to  the  TAC  ACES  I program  appear  to  be 
the  following: 

• TAC  ACES  I be  continued  as  a supplement  contingent  upon  correc- 
tion of  the  significant  deficiencies 

• Fidelity  of  target  aircraft  images  be  improved 

• All  aircraft  math  models  be  improved  to  provide  realistic  hand- 
ling qualities/performance 

• External  visual  altitude  cues  be  provided 

• Cockpit  motion  requirements  be  reviewed 

Unfortunately,  a review  of  Reference  30  did  not  indicate  pilot  comments  that 
indicated  directly  that  motion  capability  was  not  missed  or  necessary  in  the 
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training  program.  In  fact,  there  are  several  comments  indicating  potential 
problems  due  to  lack  of  appropriate  "g"  simulation,  stick  feel  during  maneuvers, 
and  inadequate  acceleration/deceleration  cues.  Nevertheless,  in  general,  the 

TAC  ACES  1 program  has  indicated  the  potential  value  of  concentrated  training  in 
ACM/BFM,  with  the  use  of  simulation  device  to  augment  switchology  skills  and 
help  develop  and  maintain  "a  sense  of  situational  awareness."  TAC  ACES  II 
was  directed  at  the  use  of  the  SAAC  simulation  facility  to  develop  an  air-to- 
air  combat  training  program  to  some  degree  built  on  the  lessons  being  learned 
during  TAC  ACES  I exercise.  While  this  program  also  indicated  limitations 
in  equipment,  it  was  considered  valuable  by  the  participants.  It  was  deter- 
mined during  this  program  that  the  "SAAC  did  not  have  the  total  fidelity  needed 
to  support  the  training  program  objective  of  quality  training."  However,  this 
document  (Reference  34)  also  states  "Every  participant  indicated  that  the  sim- 
ulator training  increased  the  value  of  the  course  beyond  that  of  one  that 
offered  only  lectures  and  discussions.  The  simulator  training  allowed  imme- 
diate reinforcement  of  classroom  discussions,  helped  develop  timing  and  hand- 
eye  coordination,  and  provided  a direct  (practical)  application  of  academic 
training."  Thus,  it  was  concluded  that  "TAC  ACES  II  provided  a beneficial 
supplement  to  current  air-to-air  training  programs."  This  was  determined  from 
pilot  program  evaluation  and  critiques.  One  recommendation  reached  on  this 
program  concerning  the  necessity  for  motion,  based  on  the  false  cues  presented 
by  the  SAAC  motion  system  is  an  over-generalization.  The  only  appropriate 
observation  is  that  no  motion  may  be  less  distracting  and  bothersome  to  the 
pilot  during  ACM  training  in  a simulator  than  the  existing  SAAC  motion  system 
and  associated  drive  logic.  Overall  utilization  of  the  SAAC  was  restricted 
due  to  facility  failures  and  maintenance  downtime  and  it  was  concluded  that 
the  36.1%  in  commission  rate  could  not  support  the  training  program  suffi- 
ciently and  that  continuation  programs  "should  not  be  scheduled  in  the  SMC 
until  maintenance  can  support  an  80%  utilization  rate."  The  other  primary 
limitations/deficiencies  that  were  identified  in  this  program  using  the  SAAC 
facility  were: 

• Visual  simulation  fidelity  needs  improvement 

• Math  model  does  not  provide  appropriate  fidelity  for  perfor- 
mance and  handling  qualities  parameters 
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• G-seat  program  needs  improvement,  "can't  quite  simulate  the 
real  feel  of  positive,  negative,  and  zero  G." 

• AOA  warning  tone  missing,  important  cue  to  provide  stall 
warning 

• Missile  envelopes  unrealistic 

• Capability  for  sun  to  conceal  target  is  highly  desirable 

• Lack  of  cue  to  alert  defender  of  weapon  employment  by  attacker 

• Record/playback  is  complicated  to  operate,  and  at  times  could 
not  provide  timely  data 

As  previously  discussed,  the  limitations  of  the  SMC  indicated  that  while  it 
could  support  a short-term  highly  controlled  training  program  that  could  be 
of  benefit,  the  device  "could  lead  to  developing  bad  flying  habits."  It  also 
appears  that  one  of  the  most  limiting  features  of  the  SMC  was  the  excessive 
maintenance  downtime.  Both  TAC  ACES  I and  TAC  ACES  II  have  indicated  the  essen- 
tially existing  state-of-the-art  technological  capability  exists  such  that 
simulation  facilities  to  supplement  current  air-to-air  training  programs  is 
realizable.  The  devices  used, even  with  their  lack  of  fidelity,  did  provide  a 
beneficial  program  to  review  and  relate  academics,  procedures  and  situation 
awareness.  The  message  also  comes  through  that  the  improvement  in  maintain- 
ability and  reliability  of  such  facilities  is  essential  to  support  any  reason- 
able training  program.  Based  on  pilot  comments  documented  in  References  30 
and  34,  the  technology  used  on  VACS  and  SMC  for  the  TAC  ACES  I and  II  program 
can  presently  provide  supplementary  training  in  a controlled  environment  but 
cannot  be  used  as  a flying  time  substitute  for  the  air-to-air  task. 

A- 10  Aircrew  Training  Device  Trade  Study 

A trade  study  jointly  prepared  by  TAC/AFSC  was  released  April  1975 
which  investigated  four  alternatives  to  aircrew  training  device  requirements. 

As  a result  of  the  various  assumptions  used  in  this  study,  it  was  determined 
that  a full  mission  simulation  capability  was  the  "minimum  satisfactory  solu- 
tion to  TAC's  stated  aircrew  training  requirements  because  it  is  the  only 
option  which  provides  simulation  of  the  operational  environment  under  which 
the  A-10  pilot  is  expected  to  perform."  The  executive  summary  continues: 
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"Given  that  the  assumptions  listed  in  this  document  are  accurate,  full  mission 
simulation  is  also  the  most  cost  effective  alternative  and  will  afford  pilots 
the  means  of  developing  and  practicing  al 1 the  tactics  and  skills  needed  to 
make  the  A- 10  an  effective  weapon  system."  The  lesson  to  be  learned  from  the 
case  history  of  a full  mission  simulation  device  is  that  while  the  concept  is 
valid  and  potentially  desirable  if  you  have  to  produce  combat  ready  aircrews 
with  minimum  time  in  aircraft;  there  exists  n£  hard  evidence  on  which  to  quanti 
fy  the  effectiveness  of  full  mission  simulation  capability,  and  that  a device 
developed  in  the  immediate  past  did  not  live  up  to  expectations.  Thus,  if  we 
are  to  learn  from  history,  it  is  of  necessity  to  review  the  assumptions  made 
and  evaluate  their  criticality.  It  must  be  remembered  that  a trade  study  is 
essentially  "crank  turning"  based  on  the  cost  equations  used.  The  solution  is 
critically  tied  to  the  assumptions  on  device  substitution,  and  device  costs 
and  device  utilization  rates.  Thus,  any  study  of  this  type  is  no  better  than 
the  assumptions  used.  In  order  to  determine  the  true  value  of  the  A- 10  trade 
study,  it  is  obviously  necessary  to  evaluate  the  risk  associated  with  the 
assumptions  used.  In  general,  based  on  the  presentation  of  the  RAF  experience 
with  the  world  of  full  mission  simulation  and  the  fidelity  deficiencies  of 
existing  devices  used  in  the  TAC  ACES  programs  as  well  as  the  maintenance 
problems  associated  with  the  operation  of  complex  simulation  facilities,  it 
becomes  questionable  as  to  the  realism  of  the  A-10  trade  study  assumptions. 
Significant  questions  arise  on  these  assumptions  that  could  challenge  the  valid 
it)  of  the  conclusions  reached,  however,  if  the  assumptions  are  correct  then 
obviously  the  conclusions  are  valid.  The  following  discussion  is  an  attempt 
to  document  a challenge  to  the  A-10  trade  study  assumptions,  based  on  the  data 
obtained  in  the  literature  survey.  This  discussion  should  not  be  considered  as 
exhaustive,  however,  the  observations  are  considered  valid.  To  some  degree, 
the  existing  state-of-the-art  was  addressed  in  the  A-10  trade  study,  however, 
the  implications  of  the  existing  deficiencies  in  the  state-of-the-art  and  the 
required/assumed  device  capabilities  did  not  receive  the  level  of  attention 
required  to  indicate  if  such  devices  are  presently  feasible  or  practical. 
Examination  of  the  device  descriptions  indicate  that  except  for  the  adaptive 
training  features,  the  HIFS  is  essentially  state-of-the-art.  The  field-of- 
view  and  visual  requirements  for  the  VS  and  FMS  are  not  considered  to  be 
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practical  at  this  point  in  time.  While  the  visual  system  essentially  appears 
to  be  a derivative  of  the  SAAC  system,  the  TAC  ACES  II  study  determined  that 
"the  fidelity  of  the  imagery  was  poor"  and  improvements  required,  at  least  in 
the  air-to-air  task.  The  value  or  level  of  fidelity  for  a ground  attack  task 
has  not  yet  been  documented  for  the  SAAC  visual  system,  to  the  best  of  our 
knowledge,  from  the  literature  survey.  Thus,  it  appears  that  extensive 
research  and  development  may  he  required  to  determine  the  minimum  level  of 
fidelity  required  for  the  visual  external  world  simulation.  To  some  degree, 
this  subject  has  been  discussed,  researched  and  reviewed  for  many  years  (i.e. 
Reference  67.  At  this  writing,  the  results  of  our  literature  survey  indi- 
cate the  existing  state-of-the-art  for  fidelity  of  visual  simulation  is  not 
adequate  for  the  intended  purpose  of  the  VS  and  FMS  devices.  In  addition,  an 
attempt  to  create  a 2vl  capability  on  SAAC  for  the  TAC  ACES  II  was  unaccept- 
able and  this  has  direct  implication  on  the  desired  tactics  training  capability 
of  the  FMS.  Thus,  before  proceeding  to  the  other  assumptions,  it  appears 
that  FMS  is  presently  not  practical  especially  from  an  operational  point  of 
view.  However,  since  we  can  assume  that  someone  will  be  willing  to  foot  the 
bill  to  do  the  required  research  and  development  in  a timely  manner  such  that 
the  desired  system  capability  becomes  practical  during  the  life  cycle  of  the 
aircraft,  there  still  remains  another  significant  problem  area.  There  still 
remains  the  subject  of  the  appropriateness  of  the  substitution  of  simulator 
hours  for  flight  hours  (transfer  effectiveness  ratio).  At  this  point,  it  should 
be  remembered  that  transfer  effectiveness  is  training  system  and  task  dependent, 
and  little  or  no  "hard"  data  is  available  except  primarily  for  instrument  and 
procedural  tasks.  Even  at  this,  the  A- 10  trade  study  tends  to  use  the  more 
optimistic,  rather  than  average  value  of  transfer  effectiveness  ratios  documented 
in  the  literature.  This  then  impacts  on  the  trade  study  by  effectively  over- 
estimating the  value  of  simulator  training  and  thereby  resulting  in  "optimis- 
tic" cost  effectiveness  for  a device,  especially  for  the  VS  and  FMS  devices. 

This  tends  to  result  in  a minimum  number  of  simulator  hours  (and  a significant 
reduction  in  flight  hours)  to  reach  a desired  proficiency  with  respect  to  not 
using  a simulator.  This  fact,  coupled  with  the  assumed  high  availability  hours 
for  the  device  (5000  per  yr)  result  in  procurement  of  minimum  number  of  devices, 
thus  minimizing  front-end  costs.  Examination  of  step  11  in  Appendix  F indicates 
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that  the  TFTW  must  achieve  essentially  5000  hours/year/cockpit  for  the  FMS. 

This  implies  an  in-commission  rate  of  essentially  100%.  If  we  can  assume  that 
the  SAAC  represents  a device  similar  to  the  desired  A-10  FMS,  during  the  TAC 
ACES  II  program  the  SAAC  facility  only  could  support  a 36.1%  in-commission  rate 
on  scheduled  hours.  This  then  indicates  another  questionable  area  with  re- 
spect to  the  A-10  Trade  Study.  The  maintainability  and  reliability  of  existing 
devices  may  be  inadequate  without  significant  product  improvement  programs  to 
support  the  required  utilization  rate.  If  as  concluded  in  Reference  34,  a de- 
sirable in-commission  rate  is  80%  to  support  a training  program,  this  factor 
should  be  "cranked"  into  the  cost-effectiveness  study,  especially  since  the  prob- 
ability of  maintenance  downtime  usually  increases  with  the  complexity  of  the 
device.  It  is  considered  unrealistic  not  to  assume  that  a device  as  complex 
and  sophisticated  as  the  A-10  FMS  will  not  be  subject  to  "Murphy's  Laws", 
especially  during  a training  session.  Another  factor  that  must  be  examined 
is  the  assumption  that  ordinance  costs  can  only  be  reduced  by  use  of  the  VS 
and  FMS  simulator  devices.  It  is  apparent  from  an  examination  of  data  con- 
cerning the  Airborne  Instrumentation  System  (AIS)  used  at  the  Air  Combat  Man- 
euvering Range,  and  data  presented  in  References  68  and  69  as  well  as  experi- 
ence gained  using  instrumentation  in  SAC  Bomb  Competitions  that  it  is  not  only 
feasible  but  within  the  state-of-the-art  to  simulate  weapons  delivery  using 
the  actual  aircraft  as  the  training  device.  The  impact  of  this  technology  can 
significantly  influence  the  A-10  Trade  Study  by  reducing  the  armament  costs 
associated  with  the  actual  aircraft  phase  directly,  and  indirectly  challenge 
the  use  of  the  simulator  for  training  the  weapons  delivery/tactics  part-tasks. 

The  last  challenge  to  the  assumptions  is  based  on  the  projected  aircraft  oper- 
ating costs,  etc.  vs.  the  simulator.  According  to  the  data  published  in  the 
A-10  Trade  Study  each  flying  hour  saved  at  the  Training  Wing  may  well  save  $429, 
while  each  flying  hour  saved  at  the  Tactical  Wing  may  well  save  $361.  The  hourly 
cost  of  the  simulator  varies,  by  using  agency  and  by  device,  with  the  range 
between  $78.48  and  $105.45  hourly  0 8 M/cockpit.  Data  published  recently  in 
U.S.  Army  Aviation  Digest,  (Reference  70)  indicates  that  the  approximate  cost 
of  one  hour  of  instrument  training  on  Device  SFTS  2B24  was  $65,  which  tends  to 
lend  credibility  to  the  estimated  operating  cost  for  the  HIFS  A-10  device. 

An  article  in  the  October  4th,  1976  issue  of  Aviation  Week  and  Space  Technology 
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by  B.  M.  Hlson  indicates  that  Boeing  figures  it  costs  $280/hr  to  use  a 727  sim- 
ulator using  a GE  visual  system.  The  display  system  uses  forward  view  and  side 
view  with  the  field  of  view  of  each  segment  30  degrees  vertically  by  40  degrees 
horizontally  and  television  tubes  capable  of  1,200  line  resolution  (2.7  arc-min 
of  angular  resolution).  While  obviously  there  are  significant  differences  be- 
tween an  A- 10  FMS  and  a 727  flight  simulation  device,  the  cost  of  operating 
the  A-10  FMS  of  approximately  $100/cockpit  hour  may  be  optimistic.  While  a 
sensitivity  study  was  performed  to  examine  increasing  fuel  costs  and  increasing 
munitions  costs,  thus  making  the  simulator  amortization  occur  somewhat  sooner, 
a sensitivity  study  was  not  performed  to  evaluate  the  impact  of  a significant 
increase  in  simulator  0 5 M costs  especially  for  the  VS  and  FMS  devices.  Ef- 
fectively, the  sensitivity  studies  that  were  performed  could  be  considered  to 
be  one-sided  to  make  the  simulator  appear  more  cost-effective  and  to  definitely 
make  option  III  (Full  Mission  capability)  more  "cost-effective ."  Thus  the 
results  of  the  Trade  Study  are  not  necessarily  "objective"  and  most  of  the 
assumptions  related  to  the  substitution  of  simulation  hours  for  flight  hours 
cannot  be  justified  by  "hard"  experimental  evidence  for  those  tasks  that  make 
the  FMS  result  in  the  most  "cost-effective"  device.  The  data  available  in  the 
literature  does  tend  to  support  the  assumptions  relative  to  the  HIFS,  assuming 
that  the  cost  per  device  of  approximately  5.7  million  is  correct.  There  is 
also  some  concern  about  the  increased  capability  costs  when  going  from  the  HIFS 
to  the  VS  to  the  BMS.  While  the  additional  features  of  the  VS  appear  to  be 
a realistic  increment  (—3.0  million)  over  the  cost  of  the  HIFS,  the  increment 
of  approximately  .55  million  to  obtain  FMS  capability  from  the  VS  device  appears 
to  be  optimistic,  especially  to  create  the  additional  visual  simulation  capa- 
bility and  cockpit  coupling  required  for  the  FMS  training  device. 

U.S.  Navy  UPT  Task  Analysis 

A recent  study,  initiated  at  the  request  of  CNET  was  performed  by 
CNATA  to  verify  current  training  procedures  and  curricula  or  to  identify  defi- 
ciencies requiring  modification  or  improvement  is  documented  in  References  71 
and  72.  The  Phase  I study  (Reference  71)  presents  data  obtained  by  question- 
naire from  experienced  fleet  aviators  and  second-tour  aviators  in  operational 
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squadrons  receiving  newly  designated  pilots  for  additional  training.  The  eval- 
uators were  required  to  rate  specified  UPT  tasks  in  terms  of  frequency,  criti- 
cality, and  training  adequacy  (similar  to  the  concept  described  by  Cream  in 
Reference  45).  Questionnaires  specific  to  particular  training  were  used  and 
the  summary  of  the  data,  in  particular,  fleet  determined  underemphasized  and 
overemphasized  training  tasks  by  community  is  presented  in  the  Phase  I report. 
Phase  II  of  the  study  then  used  the  data  determined  from  Phase  I with  appro- 
priate training  effectiveness  methodology  to  recommend  a comprehensive  program 
revision  to  obtain  fleet  desired  training  objectives  by  adjusting  the  curri- 
cula, training  techniques  and  the  devices  selected.  It  is  of  interest  to  note 
that  a study  was  initialized  to  evaluate  the  feasibility  of  using  ACMR  systems 
appropriately  modified  to  provide  training  for  various  flight  phases  other 
than  ACM.  These  training  areas  are  representative  of  the  areas  currently 
envisaged  by  the  A-10  Trade  Study  for  full  mission  simulation  capability  using 
a ground  training  device.  The  U.S.  Navy  appears  to  be  pursuing  the  unique  capa- 
bilities of  ACMR  to  provide  improved  training  and  consequent  cost  reduction. 

This  indicates  the  possibility  of  modified  ACMR  as  an  alternative  to  complex 
ground  simulation  facilities  and  the  viability  of  this  potential  should  be 
explored  in  depth.  The  results  of  the  Phase  II  study  relative  to  vhe  jet  train- 
ing requirements  and  the  recommendations  for  simulation  are  reviewed  in  the  sub- 
sequent discussion  for  the  jet  pipeline.  The  overall  result  of  Phase  II  indi- 
cates that  to  meet  fleet  validated  objectives,  an  increase  in  21  flight  hours 
and  45  simulator  hours  were  required.  The  distribution  of  the  change  in  hours 
was  as  follows: 

• Basic  — increase  flight  hours  from  26.0  to  70  and  introduce 
39.0  hours  on  simulator,  of  which  27.2  flight  hours  and  33 
simulator  hours  dedicated  to  instrument  training  (Bl/RI) . 

• Jet  Intermediate  — decrease  flight  hours  from  117.7  to  95.9 
while  increasing  simulator  time  by  21  hours,  of  which  18  sim- 
ulator hours  were  directed  to  instrument  training  with  a reduc- 
tion of  9 flight  hours. 
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• Jet  Advanced  - decrease  flight  hours  from  113  to  105.1,  de- 
crease simulator  hours  from  72  to  57,  primarily  as  a result 
of  change  in  emphasis  of  instrument  training  to  basic  and 
intermediate  training  phases. 

To  summarize  the  instrument  time  comparison,  the  current  program  requires  64.5 
flight  hours  and  78.0  simulator  hours,  while  the  proposed  program  requires  63.0 
flight  hours  and  109  simulator  hours.  Thus,  for  instrument  training  to  achieve 
the  desired  objectives  simulator  training  was  increased  by  31  hours  while  flight 
time  was  reduced  1.5  hours.  The  use  of  the  simulator  hours  for  the  various 
training  phases  is  as  follows: 

• Basic  - 39  hours  with  33  for  instrument  training  with  6 OFT 
hours  for  cockpit  procedures,  practice  normal  and  emergency 
procedures  and  a progress  check. 

• Jet  Intermediate  — 61.5  hours  with  48  for  instrument  training 
and  13.5  OFT  hours  essentially  for  normal  procedures,  emer- 
gency procedures  and  selected  malfunctions  teaching  and 
review. 

• Jet  Advanced  — 57  hours  with  28  hours  for  instrument  training, 
14  hours  in  OFT  for  procedures  and  15  hours  in  the  2F-90 
visual  simulator  (6  hours  for  familiarization,  6 hours  for 
weapons  (30°  bomb  pattern)  and  3 hours  directed  at  carrier 
qualification. ) 

It  should  be  noted  that  the  2F-90  visual  simulator  is  not  being 
used  in  this  pipeline  for  any  of  the  following  tasks: 

• night  familiarization 

• 10°  strafe 

• 30°  rocket  delivery 

• 45°  bomb  delivery 


• ACM  offensive  tactics 

• ACM  defensive  tactics 

• ACM  dissimilar  tactics 

All  of  the  above  are  trained  in  flight  rather  than  with  the  simulator,  essen- 
tially based  on  an  evaluation  of  what  can  be  trained  in  the  2F-90  visual  simu- 
lator. It  is  noted  in  the  Phase  II  document  that  several  practical  consider- 
ations were  used  in  developing  curricula  modifications,  especially  using  the 
concept  of  maximum  feasible  substitution  of  simulator  time  for  flight  time. 

This  is  illustrated  by  the  use  of  simulator  hours.  Of  the  total  of  157.5 
simulator  hours,  109  hours  are  used  for  instrument  training,  33.5  hours  are 
used  in  an  OFT  for  normal  procedures,  emergency  procedures  and  malfunctions  and 
15  hours  (less  than  10%  of  total  simulator  hours)  are  dedicated  to  the  visual 
simulator.  Based  on  the  data  identified  in  the  literature  survey  on  training 
transfer,  the  use  of  simulation  capabilities  as  outlined  in  the  Phase  II  report 
seems  quite  realistic,  and  attempts  to  reduce  costs  by  proper  utilization  of 
existing  capability.  It  is  also  interesting  to  note  that  the  Phase  II  study 
recommends  continued  study  of  the  ACMR  as  a training  tool  and  does  not  indi- 
cate or  acknowledge  a requirement  for  full  mission  simulation  capability  for 
a ground  training  facility.  As  stated  in  the  Phase  II  report  foreword,  "Imme- 
diate accommodation  to  validated  goals  is  the  study's  primary  objective." 

The  Phase  II  study  was  published  on  July  1975,  unfortunately  no  additional 
documentation  was  available  to  determine  reaction  to  the  study  recommendations 
for  modifications  to  the  various  flight  training  programs  in  the  U.S.  Navy. 
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Section  IV 


SUMMARY 


This  section  will  be  an  attempt  to  summarize  the  application  of 
flight  simulators  to  military  tactical  aircrew  training  based  on  the  data  ob- 
tained from  the  literature -survey  and  presented  in  this  report. 

While  many  who  read  this  document  may  come  away  with  a feeling 
that  this  report  is  basically  negative,  this  was  not  the  intent  of  the  study. 

It  is  highly  desirable  when  attesting  to  perform  an  objective  assessment  of 
simulation  for  the  intended  missions,  to  use  objective  data.  Unfortunately, 
little  or  no  objective  data  exist  that  would  justify  most  of  the  assumptions 
used  to  "demonstrate"  the  cost-effectiveness  of  complex  flight  training  simu- 
lators. Spending  vast  sums  of  money,  to  procure  devices  for  which  training 
effectiveness,  maintainability  and  reliability  cannot  be  demonstrated  on  even 
an  experimental  basis  is  not  necessarily  a viable  solution  for  the  military 
training  problem.  Conducting  research  using  devices  which  are  inherently  limited 
to  evaluate  the  impact  of  "cues"  that  cannot  be  adequately  represented  is  just 
not  good  "scientific"  technique.  Attributing  capability  to  devices  to  indicate 
that  they  have  "ultra-realism"  may  be  good  for  press  releases,  but  should  be 
relegated  to  the  area  of  wishful  thinking  rather  than  technically  feasible, 
practical  or  even  achievable  based  on  the  existing  state-of-the-art.  On  the 
other  hand,  that  training  devices  can  be  effectively  used  to  reduce  flight  time, 
has  been  documented,  both  by  the  Airlines  and  the  Military  in  several  reports, 
but  primarily  only  in  the  training  of  procedures  (both  normal  and  emergency) , 
malfunctions,  general  familiarization,  and  instrument  flight  training.  While 
little  transfer  effectiveness  data  exist,  there  does  appear  to  be  a general 
rule  of  thumb  emerging  from  the  available  data  that  "simulator  time  increases 
about  twice  as  fest  as  airplane  time  decreases."  In  addition,  recent  programs 
have  indicated  that  supplementary  training  in  complex  tasks,  even  using  devices 
of  limited  fidelity,  is  practical.  Thus  it  appears  that  to  minimize  cost,  a 
viable  short-term  approach  would  be  to  maximize  the  use  of  the  simulation 
facilities  in  those  areas  in  which  they  have  demonstrated  training  effectiveness 
and  to  develop  this  capability  to  maximize  "cost-effectiveness"  realistically. 

In  addition,  cost  savings  can  be  achieved  by  minimizing  the  undergraduate  pilot 
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washout  rate  by  using  more  effective  screening  procedures.  Alternatives  to  the 
use  of  simulation  to  conserve  limited  fuel  resources  and  reduce  ordinance  costs 
are  feasible,  and  a more  careful  assessment  is  required  of  the  following  areas: 

• Fuel  efficient  engine  development. 

• Use  of  optimal  control  theory  to  develop  fuel  efficient 
aircraft  profiles  both  for  operational  and  training 
missions. 

• Development  of  low  cost  airborne  instrumentation  systems 
that  can  be  used  to  simulate  weapons  delivery  using  an 
aircraft  and  be  recorded  for  future  review  by  students. 

• Development  of  relatively  low  cost,  in-flight  simulators 
for  part-task  training  purposes  using  existing  state- 
of-the-art  technology. 

In  addition,  and  possibly  most  important,  is  the  need  to  determine 
minimum  device  requirements  to  obtain  the  desired  training  objectives.  Skill 
retention  studies  indicate  that  procedural  skills  are  the  most  easily  forgotten 
and  fortunately  this  is  an  area  where  part-task  simulation  devices  can  be  used 
effectively.  Data  have  not  been  determined  that  indicate  how  simulators  can 
enhance  skill  level,  especially  since  the  1SD  concept  is  to  train  to  proficiency. 
Whether  or  not  the  way  these  devices  are  used  is  also  the  most  efficient  re- 
quires additional  research  by  the  training  psychologists.  However,  the  first 
order  of  business  is  to  be  able  to  describe  how  the  pilot  functions  in  the  task, 
and  that  answer  can  only  be  found  in  the  air,  not  by  using  ground  based  flight 
simulation  facilities. 

There  are  also  questions  raised  about  the  maintainability  and 
reliability  of  flight  simulation  devices  and  product  improvement  programs  such 
as  those  indicated  recently  in  many  journals  by  Link-Singer  must  b~  encouraged 
to  obtain  the  full  utilization  potential  from  the  training  simulator.  Thus, 
while  substitution  is  possible  in  certain  specific  training  areas,  only  sup- 
plemental training  using  simulators  is  presently  practical  for  most  tactical 
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applications.  Significant  additional  research  is  required  to  be  capable  of 
justifying  complex  full  mission  simulators  on  the  basis  of  "objective"  data. 

For  the  tactical  missions  to  be  trained,  it  becomes  obvious  tnat 
a fundamental  requirement  is  to  determine  what  minimum  level  of  visual  fidelity 
is  necessary  to  accomplish  the  desired  training  objective  and  to  compare  those 
research  and  development  costs  with  the  possible  alternative  of  the  develop- 
ment of  low-cost  airborne  weapons  simulation  capability.  As  discussed  by 
Captain  John  E.  Richardson  in  Reference  73,  concerning  the  use  of  the  ACMR , 

"it  is  vital  that  every  hour  of  flying  time  be  productive."  The  use  of 
facilities  such  as  ACMR,  the  continued  development  of  the  Bomb  Scoring  System 
discussed  in  Reference  68,  and  the  WASI  concept  prescribed  in  Reference  69 
represent  alternatives  for  realistic,  efficient  and  co  -effective  utilization 
of  flying  time  for  tactical  training. 

Another  area  that  must  be  evaluated  is  the  proper  training  and 
utilization  of  instructors  and  console  operators  for  training  devices.  The 
literature  acknowledges  that  even  the  best  device  can  be  compromised  and 
rendered  ineffective  and  inefficient  by  improperly  trained  instructors.  Thus, 
before  a device  can  be  used  to  train  students,  it  is  vital  to  qualify  the 
instructor  and  provide  the  proper  promotional  incentives  to  retain  capable 
instructors . 


Significant  problems  exist  in  the  determination  of  "objective"  mea- 
sures of  performance  that  can  be  used  to  validate  the  effectiveness  of  a training 
system  or  the  skill  level  of  the  trainee.  At  present,  the  technique  appears 
to  be  to  define  a performance  basis,  measure  deviations  from  the  basis,  obtain 
statistical  measures  of  the  error  and  then  "massage"  the  data  to  obtain  correla- 
tion with  the  "subjective"  evaluations  of  an  instructor.  This  approach  cannot 
truly  be  considered  "objective"  since  at  almost  each  decision  point  subjective 
judgments  are  required,  for  example,  defining  the  performance  basis  for  a complex 
task,  defining  important  errors  and  their  appropriate  thresholds,  statistical 
analysis  of  errors  rather  than  frequency  analysis,  assuming  data  is  amenable 
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to  statistical  analysis  and  finally  using  regression  analysis  to  curve  fit  to 
the  "subjective"  instructor  rating.  Obviously,  this  last  step  puts  a lot  of 
credibility  into  the  "subjective"  approach  so  perhaps  it  may  be  more  'cost- 
effective'  to  provide  a well-defined  subjective  rating  scale,  sensitive  to  the 
training  objectives,  than  to  put  too  much  credibility,  at  present,  into 
"objective"  performance  measurement.  In  addition,  the  literature  indicates 
that  pilots  prefer  to  be  evaluated  by  their  peers  rather  than  by  a computer. 

It  is  important  to  evaluate  what  motivates  the  professional  military 
pilot  that  would  encourage  him  to  accept  simulator  training  as  important,  es- 
pecially to  his  survival  in  a combat  environment.  Giving  the  pilots  flight  pay 
for  flying  a simulator  may  be  an  interesting  "carrot"  to  some,  and  obviously 
would  not  be  rejected  by  all  given  no  other  choice.  However,  if  the  use  of  th  • 
simulator  will  not  maintain  the  skills  required,  or  improve  skills  to  maintain 
or  train  aircrews  to  a combat  ready  status,  then  "carrot"  dangling  may  not  be 
sufficiently  motivating  to  professionals. 

The  primary  question  still  to  be  addressed,  aside  from  the  money 
issue,  is  the  total  impact  of  using  flight  simulators  on  the  combat  effective- 
ness of  the  entire  force.  As  stated  by  Dr.  Allen  in  Reference  1,  "This  effec- 
tiveness is  dependent  on  a total  team  effort.  Equipment,  aircrews  and  ground 
support  are  integral  parts  of  any  successful  combat  operation.  This  integra- 
ted effort  cannot  be  exercised  through  flight  simulation."  It  should  be  noted 
that  some  of  the  alternatives  discussed  to  the  use  of  ground-based  devices  to 
train  aircrews  can  exercise  the  total  system. 
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